Site Title

Caching

March 17, 2018 sajintecblog

Types of caching strategies:

From <https://msdn.microsoft.com/en-us/magazine/hh708748.aspx>

Usecase for caching:

The content is expensive to produce. Fetching a web page and rendering it can take a few seconds each time.

The content can’t be precomputed. The number of web pages is practically infinite, and there’s no way to predict which ones will be requested (and when).

The content changes relatively slowly. Most web pages don’t change second by second, so it’s okay to serve up an old thumbnail. (In my app, I chose five minutes as the threshold for how old was acceptable.)

Types of Caching:

Data caching-If some dataset has to be repeated in multiple views,

Output caching-If a view has output generated in a complicated way, cache that output. It is the content returned by a controller action. By setting Location attribute, we can define if it has to be cached in server/client/any/downstream. VaryByParam will determine if content has to be varied according to a key.

https://www.asp.net/mvc/overview/older-versions-1/controllers-and-routing/improving-performance-with-output-caching-cs

Javascript tools used for better caching:

MeteorJS, Firebase, asana Luna.

Resource Caching

In-Memory Caching

Distributed Cache:

Factors to check for distributed caching:

Temporal
Spatial
Transactional
Cache Replacement(LRU,MRU,LFU)
Primed vs Demand
Shared(processed or clustered
Caching size(Pareto principle, that is 80-20 principle)

How to update data which is already caches

What is Etag in HTTP?

Using the CachingHandler in ASP.NET Web API:

http://byterot.blogspot.in/2012/06/aspnet-web-api-caching-handler.html

https://github.com/sajints/CacheCow

Distributed Caching:

https://vimeo.com/album/2683752/video/83758187

Cache-Aside Indirection

What is distributed caching?

https://msdn.microsoft.com/en-us/library/dd129907.aspx

Redis eviction policies:

Maxmemory-policy configuration value is driving the eviction from memory. Redis checks and based on this property value, memory will be evicted.

Noeviction – return error when maximum memory is reached
Allkeys-lru – LRU keys will be removed first
Volatile-lru -LRU keys will be removed first, only for those keys which has expiry set
Allkeys-random – evict random keys
Volatile-random – evict random keys, which has expiry set
Volatile-ttl – evict based on TTL

Amazon ElastiCache

ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory caches, instead of relying entirely on slower disk-based databases. ElastiCache supports two open-source in-memory caching engines:

Memcached – a widely adopted memory object caching system. ElastiCache is protocol compliant with Memcached, so popular tools that you use today with existing Memcached environments will work seamlessly with the service.
Redis – a popular open-source in-memory key-value store that supports data structures such as sorted sets and lists. ElastiCache supports Master / Slave replication and Multi-AZ which can be used to achieve cross AZ redundancy.

Amazon ElastiCache automatically detects and replaces failed nodes, reducing the overhead associated with self-managed infrastructures and provides a resilient system that mitigates the risk of overloaded databases, which slow website and application load times. Through integration with Amazon CloudWatch, Amazon ElastiCache provides enhanced visibility into key performance metrics associated with your Memcached or Redis nodes.

Using Amazon ElastiCache, you can add an in-memory caching layer to your infrastructure in a matter of minutes by using the AWS Management Console.

From <https://aws.amazon.com/elasticache/>

What is cache-aside pattern?

https://msdn.microsoft.com/en-us/library/dn589799.aspx

Cache-Aside Pattern

Load data on demand into a cache from a data store. This pattern can improve performance and also helps to maintain consistency between data held in the cache and the data in the underlying data store.

Context and Problem

Applications use a cache to optimize repeated access to information held in a data store. However, it is usually impractical to expect that cached data will always be completely consistent with the data in the data store. Applications should implement a strategy that helps to ensure that the data in the cache is up to date as far as possible, but can also detect and handle situations that arise when the data in the cache has become stale.

Solution

Many commercial caching systems provide read-through and write-through/write-behind operations. In these systems, an application retrieves data by referencing the cache. If the data is not in the cache, it is transparently retrieved from the data store and added to the cache. Any modifications to data held in the cache are automatically written back to the data store as well.

For caches that do not provide this functionality, it is the responsibility of the applications that use the cache to maintain the data in the cache.

An application can emulate the functionality of read-through caching by implementing the cache-aside strategy. This strategy effectively loads data into the cache on demand. Figure 1 summarizes the steps in this process.

Figure 1 – Using the Cache-Aside pattern to store data in the cache

If an application updates information, it can emulate the write-through strategy as follows:

Make the modification to the data store
Invalidate the corresponding item in the cache.

When the item is next required, using the cache-aside strategy will cause the updated data to be retrieved from the data store and added back into the cache.

From <https://msdn.microsoft.com/en-us/library/dn589799.aspx>

Testing

March 17, 2018March 17, 2018 sajintecblog

What is Coded UI testing?

Automated tests that drive your application through its user interface (UI) are known as coded UI tests (CUITs).

These tests include functional testing of the UI controls. They let you verify that the whole application, including its user interface, is functioning correctly. Coded UI Tests are particularly useful where there is validation or other logic in the user interface, for example in a web page. They are also frequently used to automate an existing manual test.

https://msdn.microsoft.com/en-us/library/dd286726.aspx#VerifyingCodeUsingCUITCreate

How to register classes in unity?

RegisterTypes:

Method	Description	Platform
FromAssemblies	Takes as parameters the list of assemblies to scan. Returns all visible, non-abstract classes from this list of assemblies. Optionally, control how exceptions are handled when getting types from an assembly by using theskipOnError parameter.	Desktop and Windows Store apps
FromAssembliesInBasePath	Returns all visible, non-abstract classes from all assemblies located in the base folder of the current application domain. Optionally, control how exceptions are handled when getting types from an assembly by using theskipOnError parameter. Optionally control whether to load system assemblies, dynamic assemblies, and Unity assemblies by using the includeSystemAssemblies, includeDynamicAssemblies, andincludeUnityAssemblies parameters.	Desktop
FromLoadedAssemblies	Returns all visible, non-abstract classes from all assemblies loaded in the current application domain. Optionally, control how exceptions are handled when getting types from an assembly by using the skipOnError parameter. Optionally control whether to load system assemblies, dynamic assemblies, and Unity assemblies by using theincludeSystemAssemblies, includeDynamicAssemblies, and includeUnityAssemblies parameters.	Desktop
FromApplication	Returns all visible, non-abstract classes from all assemblies loaded in the install location of the application. Optionally, control how exceptions are handled when getting types from an assembly by using the skipOnErrorparameter. Optionally control whether to load system assemblies, dynamic assemblies, and Unity assemblies by using the includeSystemAssemblies, includeDynamicAssemblies, and includeUnityAssemblies parameters.	Windows Store apps

Unity Lifetime Managers:

The Unity container manages the creation and resolution of objects based on a lifetime you specify when you register the type of an existing object or on the default lifetime if you do not specify a lifetime manager for it to use.

When you register a type using the RegisterType method, the default behavior is for the container to use a transient lifetime manager. It creates a new instance of the registered, mapped, or requested type each time you call the Resolve or ResolveAll method or when the dependency mechanism injects instances into other classes. The container does not store a reference to the object. However, when you want singleton behavior for objects Unity creates, the container must store a reference to these objects. It also takes over management of the lifetime of the objects.

Unity uses specific types that inherit from the LifetimeManager base class (collectively referred to as lifetime managers) to control how it stores references to object instances and how the container disposes of these instances.

When you register an existing object using the RegisterInstance method, the default behavior is for the container to take over management of the lifetime of the object you pass to this method using the ContainerControlledLifetimeManager. This means that the existing object remains in scope as long as the container is in scope, and it is disposed when the container goes out of scope and is garbage-collected or when code explicitly disposes the container. You can also use this lifetime manager with the RegisterType method to specify that Unity should manage the object as a singleton instance.

The Unity Application Block includes three lifetime managers that you can use directly in your code, but you can create your own lifetime managers to implement specific lifetime scenarios. Unity includes the following lifetime managers:

ContainerControlledLifetimeManager. Unity returns the same instance of the registered type or object each time you call the Resolve or ResolveAll method or when the dependency mechanism injects instances into other classes. This lifetime manager effectively implements a singleton behavior for objects. Unity uses this lifetime manager by default for the RegisterInstance method if you do not specify a different lifetime manager. If you want singleton behavior for an object that Unity creates when you use the RegisterType method, you must explicitly specify this lifetime manager. The behavior is as follows:
- If you used the RegisterType method to register a type, Unity creates a new instance of the registered type during the first call to the Resolve or ResolveAll method or when the dependency mechanism injects instances into other classes. Subsequent requests return the same instance.
- If you used the RegisterInstance method to register an existing object, Unity returns this instance every time you call the Resolve or ResolveAll method or when the dependency mechanism injects instances into other classes.
ExternallyControlledLifetimeManager. This lifetime manager allows you to register type mappings and existing objects with the container so that it maintains only a weak reference to the objects it creates when you call the Resolve or ResolveAll method or when the dependency mechanism injects instances into other classes based on attributes or constructor parameters within that class. Unity returns the same instance of the registered type or object each time you call the Resolve or ResolveAll method or when the dependency mechanism injects instances into other classes. However, the container does not hold onto a strong reference to the object after it creates it, which means that the garbage collector can dispose of the object if no other code is holding a strong reference to it.
PerThreadLifetimeManager. Unity returns, on a per-thread basis, the same instance of the registered type or object each time you call the Resolve or ResolveAll method or when the dependency mechanism injects instances into other classes. This lifetime manager effectively implements a singleton behavior for objects on a per-thread basis. PerThreadLifetimeManager returns different objects from the container for each thread. The behavior is as follows:
- If you used the RegisterType method to register a type, Unity creates a new instance of the registered type the first time the type is resolved in a specified thread, either to answer a call to the Resolve or ResolveAll methods for the registered type or to fulfill a dependency while resolving a different type. Subsequent resolutions on the same thread return the same instance.
- Using the RegisterInstance method to register an existing object results in the same behavior as if you just registered the lifetime container with RegisterType. Therefore, it is recommended that you do not use the RegisterInstance method to register an existing object when using the PerThreadLifetimeManager.
- PerThreadLifetimeManager returns the object desired or permits the container to create a new instance if no such object is currently stored for the current thread. A new instance is also created if called on a different thread than the one that set the value. This lifetime manager does not dispose the instances it holds.

https://msdn.microsoft.com/en-us/library/ff647854.aspx

Big Data

February 7, 2018 sajintecblog

Spark in .Net – Mobius

https://databricks.com/blog/2016/08/03/developing-apache-spark-applications-in-net-using-mobius.html

Algorithms

November 25, 2017 sajintecblog

- 1. Arrays and Strings
  2. Linked List
  3. Stack and Queue
  4. Trees and Graphs
  5. Bit manipulation
  6. Brain teasers
  7. Mathematics and Probability
  8. Object Oriented Design
  9. Recursion and Dynamic programming
  10. Scalability and memory limits
  11. Sorting and searching
  12. Testing
  13. Databases
  14. Threads & Locks

Data types: Stack, Queue, Bag, Union-Find, Priority-Queue

Sorting: Quicksort, MergeSort, HeapSort, RadixSort

Searching – BST, Red-Black BST, Hash table

Graphs – BFS, DFS, Prim, Kruskal, Dijkstra

Strings: KMP, Regular Expressions, TST, Huffman, LZW

Advanced: B-Tree, Suffix Array, Max flow

What is Algorithm?

Method to solve a problem

What is Data structure?

Method to store information

Reflexive: p is connected to p

Symmetric: p connected to q, then q connected to p

Transitive: p connected to q, q connected to r, then p connected to r

Algorithm Coursera study:

https://www.coursera.org/learn/introduction-to-algorithms/lecture/EcF3P/quick-find

Imp Data structures: Linked List, Binary tree, Tries, Stacks, Queues, Vectors/Array lists, Hash tables

Imp Algorithms – Breadth First Search, Depth First Search, Binary Search, Merge Sort, Quick Sort, Tree Insert/Find etc

Concepts – Bit manipulation, Singleton design pattern, Factory design pattern, Memory(stack vs heap), Recursion, Big-O Time

Algo question Approaches:

Examplify: Write specific examples of problem and see if we can derive a general rule/logic from that. Ex: Given a time, calculate angle between hour and minute hands.

Pattern Matching: See what problems are similar to this. For ex:, a sorted array is rotated in the order 3,4,5,6,1,2. How to find min number? There are 2 problems that will come in mind – Find min amount in array and Find particular element in sorted array. Binary search is useful here by comparing MID with LAST and find which is big.

Simplify and Generalize: With this we implement a multi-step approach. First we change a constraint such as data type or amount of data. Doing this will simplify the problem. Once we have an algorithm for simplified problem, try to adapt earlier solution for more complex version. Ex: A ransom note can be formed by cutting words out of a magazine. To form a new sentence How to find is a ransom note can be formed from a given magazine. To solve this, create an array and counting characters. When we generalize, rather than creating an array with character counts, we create a hash table that maps from a word to its frequency.

Base case and Build: This is great solution for some problems. We solve the problem for a base case(n=1). Then try same solution for n=2, then n=3 until n-1 and n. Ex: Design an algorithm to print all permutations of a string. Ex: abcdefg. F(a) = {“a”}, F(ab)={“ab”,”ba”}, F(abc) can be derived by inserting c to all locations of combinations of previous answer. This combining with recursive algorithms will make a suitable solution.

Data structure brainstorm: Run through a list of data structures and apply. Ex: Numbers are stored into an expanding array. How will you keep track of median? Which data structures from – Linked List, Array, Binary tree, Heap

Other Tips:

Use data structures generously

Appropriate code reuse

Modular

Flexible and robust

Error checking

Arrays and strings:

- 1. Hash table
  2. Array List
  3. StringBuffer(String Buffer)

Linked Lists:

Single Linked List

Doubly Linked List

Running technique: Two pointers, one faster and other normal.

Recursive problems:

Solve this if u r genius One person had a habit of spending money according to dates. i. e. If date is 19 he was spending Rs.19 and if date is 15, he was spending Rs. 15. One night he calculated total spending of 5 consecutive days – Monday to Friday and he found that he spent Rs. 63 in 5 days. Identify dates. For all geniuses in the groupChat Conversation End

Read more: Solve this if u r genius One person had a habit of spending money according to dates. i. e. If date is 19 he was spending Rs.19 and if date is 15,… – Solve this if u r genius One person had a habit of spending money according to dates. i. e. If date is 19 he was spending Rs.19 and if date is 15, he was spending Rs. 15. One night he calculated total spending of 5 consecutive days – Monday to Friday and he found that he spent Rs. 63 in 5 days. Identify dates. For all geniuses in the group.

28,29,1,2,3

ML

November 25, 2017 sajintecblog

5 questions ML can answer: https://docs.microsoft.com/en-us/azure/machine-learning/machine-learning-data-science-for-beginners-the-5-questions-data-science-answers

Classfication – Whether this belongs to A or B or C
Anomaly detection – Find the odd ones through anomaly detection algorithms(Un supervised, Supervised and semi-supervised)
Regression – Find how much or How many using regression algorithms
Clustering algorithms – How is this organized?Which users like same type of movies? Organize data, we can better understand and predict behaviors and events.
Reinforcement learning algorithms – What should I do now? These study the outcomes of events and helps decide actions based on it. Ex: in a temperature control system, adjust it up or down or leave it as it is. Gather information through trial and error.

How to check if the data is ready?

That can be known from – Is the data: Relevant, Connected, Accurate, Enough to work with?

Assignment:

A <- 2

Data Types:

Numeric – 1

Integer – 1

Character – “”

Logical – TRUE/FALSE

Factor – categorical ex: factor (c (“male”,”female”))

Other operations:

Ceiling, round, floor, abs,

class(q) will give data type of q.

Vectors:

Collection of elements. Ex: a <- c(1,2,4,5,”abc”) . Typically, all elements in vector will be single type. If character and numeric and mixed, character will be taken as data type for all.

Length(a) will give how many elements in a. a[3] will give 3rd position of a. The positioning starts from 1, not 0

A[5:7] will give all elements from 5 to 7. Also, a[5:length(a)] if it has to be printed till end.

If we need 1,5 and last elements of array, so write a[c(1,5,length(a))] – here c stands for concatenation

To get all variable names, ls() command to use. To remove all these, use rm(list = ls())

To get current working directory- getwd() and to set, use setwd(“c:/users”) or setwd(“c:\\users”)

A <- 1:10 will save 1 to 10 in A, same applies to descending order as well

To make a sequence, seq(1,30,2) will give values 1,3,5… It will set values incrementing by 2. seq(1, by=2, length = 50) will store exact 50 values from 1 incrementing by 2. rep(2,5) will repeat 2 – 5 times