tensorflow high level api vs low level api - tensorflow

The question is simple.
Should one go for low level APIs or high level API in tensorflow in his/her own projects?
If I go for high level APIs only in my Neural Network implementation: Will I e.g. be able to apply such solutions like dropout, batch normalization, learning decay, etc.? Or is it inevitable to start using lower level APIs for such cases?
My view so far is that the tensorflow community does a great job in almost everything they do but I unfortunately cannot say the same thing for their documentation. It really takes time to clearly figure out how they thought when implementing the API. Then the question is if it is really worth the time I and many others spend to make things work? Or should we always search for solutions in the high level APIs?
It is not clear to me either where the boundary goes between high level APIs and low level APIs.

Depends on your project goal, project novelty and complexity.
If goal of your specific project is to learn and explore tensorflow, then I would say go for low level API, because in future you would probably need to implement your stuff, since the primary purpose of tensorflow is research.
Otherwise If its team project or a serious project, then use whatever that suits the requirements, for example if existing high level solutions like layers etc.. are not working, then you would want to implement your own stuff.
So if you are learning tensorflow and starting out go for the high-level API and if you already have some experience then start exploring the low level tensorflow API as it is highly likely that you will need low-level tensorflow API in future.

Related

Which kinds of high level API of tensorflow should I learn?

I have studied tensorflow for about one month. I just feel that creating a network with primitive operations of Tensorflow is very verbose. Then I found some high level API, such as TF-Slim, TF Learn, Keras. But multiple choices confuse me so that I don't know which I should learn.
TF-Slim is a lightweight library for defining, training and evaluating complex models in TensorFlow, but as I investigated, it's only for convnets. What networks Keras can build are more diverse.
Can Anyone give a comparision between them so that I could choose which high level API I should learn ? In terms of :
1. popularity: which ones are the most popular ?
2. practicality: what kinds of network can they build ?
3. performance: what's their training/inference performance ?
... something else
Hope someone could give me a suggestion. Thanks.
I suggest you start with Keras.
It´s very easy to learn, it has a broad user base (see Shobhits link), there is a ton of reference code out there on GitHub and in tutorials / MOOCs / eBooks etc. and you can build almost anything with it. And I personally think that is has a good documentation (although some might disagree with that...).
Since it´s an API that connects to Tensorflow, Theano, CNTK (and possibly more frameworks in the future) you have even more flexibility.
Don´t worry too much about performance. That´s really not important while youre learning.

Most performant Bayesian Network Java API available?

I'm currently planning the implementation of a bayesian network solution for inference of outcome probabilities given known node networks, in a Java application. I've been looking around the web for what Java APIs are available, and have come across a number of these – jSMILE, AgenaRisk, JavaBayes, netica-J, Jayes, WEKA(?), etc.
Now, I'm struggling to find any good comparison of/compare all these APIs in terms of performance, usability and price for commercial applications, to know which one is the best to go for. I have tested out the AgenaRisk API, which suited my needs however before committing to this product it'd be great to hear if anyone has any knowledge of:
variation in performance between different APIs (or is it negligible? i.e. they all rely on identical fundamental Bayesian calculations?)
robust free alternatives to AgenaRisk?
does it matter if one of these solutions seems to no longer be supported/relies on a very old version of Java (e.g. JavaBayes is on Java 1.1 I believe)?
(Bonus points) are Bayesian Networks and Bayesian Network Classifications one and the same thing? As for example WEKA advertises itself as providing the latter.
List item
The last post on here looking for a good solution was from 2012, so I'm wondering if anyone would recommend any new solutions which have emmerged or if it's still a good bet to work with those.
Thanks!
Well for anyone who comes across this and is interested:
Will be figuring this out myself soon and will update this (If I forget nudge me).
Jayes seems to do everything AgenaRisk does if you're simply creating a graph and want to get some beliefs out of it, minus the GUI. Also it seems there you can choose what algorithm to use, which I haven't seen with AgenaRisk.
I'm going to stick to current solutions to be safe
...yes they are

What is Google's business model for Tensorflow? also why did Baidu develop PaddlePaddle

I heard Google has over 100 engineers working on Tensorflow. So, what does Google gain from distributing this expensive engine for free? Will Google later charge corporate Tensorflow users? ?
What is google’s plan for Tensorflow? will it be free forever? like a deep learning version of python?
Some people tell me it's to reduce new employee training time. If so, why did Baidu develop it's own deep learning library when that means they should spend extra time training it's employees who are used to using Tensorflow, how to use Baidu's deep learning library
Well if everyone is using tensorflow there are more contributions made to it by the public. That helps google for free.
Even if it doesn't become a product that they sell they can still make money from it.
The thing is they might be able to sell services around it like lessons. They can also sell hardware specifically optimized for tensorflow etc.
Almost every company had its own machine learning framework. The reason they open sourced them was because each of theirs were going to fall much behind if it wasn't open source. A prime example is how Torch was doing so great up until tensorflow became open source as well. (With google's backing and because it's in python instead of lua are a few reasons tf became more popular).

Fastest way to run recurrent neural network (inference) on mobile device

What I have: A trained recurrent neural network in Tensorflow.
What I want: A mobile application that can run this network as fast as possible (inference mode only, no training).
I believe there are multiple ways how I can accomplish my goal, but I would like you feedback/corrections and additions because I have never done this before.
Tensorflow Lite. Pro: Straight forward, available on Android and iOS. Contra: Probably not the fastest method, right?
TensorRT. Pro: Very fast + I can write custom C code to make it faster. Contra: Used for Nvidia devices so no easy way to run on Android and iOS, right?
Custom Code + Libraries like openBLAS. Pro: Probably very fast and possibility to link to it on Android on iOS (if I am not mistaken). Contra: Is there much use for recurrent neural networks? Does it really work well on Android + iOS?
Re-implement Everything. I could also rewrite the whole computation in C/C++ which shouldn't be too hard with recurrent neural networks. Pro: Probably the fastest method because I can optimize everything. Contra: Will take a long time and if the network changes I have to update my code as well (although I am willing to do it this way if it really is the fastest). Also, how fast can I make calls to libraries (C/C++) on Android? Am I limited by the Java interfaces?
Some details about the mobile application. The application will take a sound recording of the user, do some processing (like Speech2Text) and output the text. I do not want to find a solution that is "fast enough", but the fastest option because this will happen over very large sound files. So almost every speed improvement counts. Do you have any advice, how I should approach this problem?
Last question: If I try to hire somebody to help me out, should I look for an Android/iOS-, Embedded- or Tensorflow- type of person?
1. TensorflowLite
Pro: it uses GPU optimizations on Android; fairly easy to incorporate into Swift/Objective-C app, and very easy into Java/Android (just adding one line in gradle.build); You can transform TF model to CoreML
Cons: if you use C++ library - you will have some issues adding TFLite as a library to your Android/Java-JNI (there is no native way to build such library without JNI); No GPU support on iOS (community works on MPS integration tho)
Also here is reference to TFLite speech-to-text demo app, it could be useful.
2. TensorRT
It uses TensorRT uses cuDNN which uses CUDA library. There is CUDA for Android, not sure if it supports the whole functionality.
3. Custom code + Libraries
I would recommend you to use Android NNet library and CoreML; in case you need to go deeper - you can use Eigen library for linear algebra. However, writing your own custom code is not beneficial in the long term, you would need to support/test/improve it - which is a huge deal, more important than performance.
Re-implement Everything
This option is very similar to the previous one, implementing your own RNN(LSTM) should be fine, as soon as you know what you are doing, just use one of the linear algebra libraries (e.g. Eigen).
The overall recommendation would be to:**
try to do it server side: use some lossy compression and serverside
speech2text;
try using Tensorflow Lite; measure performance, find bottlenecks, try to optimize
if some parts of TFLite would be too slow - reimplement them in custom operations; (and make PR to the Tensorflow)
if bottlenecks are on the hardware level - goto 1st suggestion

Tensorflow - Why are there so many similar or even duplicate functions in tf.nn and tf.layers / tf.losses / tf.contrib.layers etc?

In Tensorflow (as of v1.2.1), it seems that there are (at least) two parallel APIs to construct computational graphs. There are functions in tf.nn, like conv2d, avg_pool, relu, dropout and then there are similar functions in tf.layers, tf.losses and elsewhere, like tf.layers.conv2d, tf.layers.dense, tf.layers.dropout.
Superficially, it seems that this situation only serves to confuse: for example, tf.nn.dropout uses a 'keep rate' while tf.layers.dropout uses a 'drop rate' as an argument.
Does this distinction have any practical purpose for the end-user / developer?
If not, is there any plan to cleanup the API?
Tensorflow proposes on the one hand a low level API (tf., tf.nn....), and on the other hand, a higher level API (tf.layers., tf.losses.,...).
The goal of the higher level API is to provide functions that greatly simplify the design of the most common neural nets. The lower level API is there for people with special needs, or who wishes to keep a finer control of what is going on.
The situation is a bit confused though, because some functions have the same or similar names, and also, there is no clear way to distinguish at first sight which namespace correspond to which level of the API.
Now, let's look at conv2d for example. A striking difference between tf.nn.conv2d and tf.layers.conv2d is that the later takes care of all the variables needed for weights and biases. A single line of code, and voilà, you just created a convolutional layer. With tf.nn.conv2d, you have to take declare the weights variable yourself before passing it to the function. And as for the biases, well, they are actually not even handled: you need to add them yourself later.
Add to that that tf.layers.conv2d also proposes to add regularization and activation in the same function call, you can imagine how this can reduce code size when one's need is covered by the higher-level API.
The higher level also makes some decisions by default that could be considered as best practices. For example, losses in tf.losses are added to the tf.GraphKeys.LOSSES collection by default, which makes recovery and summation of the various component easy and somewhat standardized. If you use the lower level API, you would need to do all of that yourself. Obviously, you would need to be careful when you start mixing low and high level API functions there.
The higher-level API is also an answer to a great need from people that have been otherwise used to similarly high-level function in other frameworks, Theano aside. This is rather obvious when one ponders the number of alternative higher level APIs built on top of tensorflow, such as keras 2 (now developed exclusively for tensorflow), slim (in tf.contrib.slim), tflearn, tensorlayer, and the likes.

Resources