Страница телеграм канала Spark in me - Internet, data science, math, deep learning, philosophy
Spark in me - Internet, data science, math, deep learning, philosophy
1332 подписчиков
All this - lost like tears in rain. Internet, data science, math, deep learning, philosophy. No bs. Our website - http://spark-in.me Our chat - https://goo.gl/WRm93d DS courses review - http://goo.gl/5VGU5A - https://goo.gl/YzVUKf
Входит в категории: Технологии
22.04.2019 14:04
2019 DS / ML digest 9
Highlights of the week
- Stack Overlow survey;
- Unsupervised STT (ofc not!);
- A mix between detection and semseg?;
https://spark-in.me/post/2019_ds_ml_digest_09
#digest
#deep_learning
22.04.2019 14:04
Cool docker function
View aggregate load stats by container
https://docs.docker.com/engine/reference/commandline/stats/
#linux
17.04.2019 11:04
Archive team ... makes monthly Twitter archives
With all the BS with politics / "Russian hackers" / Arab spring - twitter how has closed its developer API.
No problem.
Just pay a visit to archive team page
https://archive.org/details/twitterstream?and[]=year%3A%222018%22
Donate them here
https://archive.org/donate/
#data_science
#nlp
#nlp
17.04.2019 11:04
Using snakeviz for profiling Python code
Why
To profile complicated and convoluted code.
Snakeviz is a cool GUI tool to analyze cProfile profile files.
https://jiffyclub.github.io/snakeviz/
Just launch your code like this
python3 -m cProfile -o profile_file.cprofile
And then just analyze with snakeviz.
GUI
They have a server GUI and a jupyter notebook plugin.
Also you can launch their tool from within a docker container:
snakeviz -s -H 0.0.0.0 profile_file.cprofile
Do not forget to EXPOSE necessary ports. SSH tunnel to a host is also an option.
#data_science
14.04.2019 09:04
PyTorch DataParallel scalability
TLDR - it works fine for 2-3 GPUs.
For more GPUs - use DDP.
https://github.com/NVIDIA/sentiment-discovery/blob/master/analysis/scale.md
https://github.com/SeanNaren/deepspeech.pytorch/issues/211
#deep_learning
09.04.2019 09:04
2019 DS / ML digest number 8
Highlights of the week
- Transformer from Facebook with sub-word information;
- How to generate endless sentiment annotation;
- 1M breast cancer images;
https://spark-in.me/post/2019_ds_ml_digest_08
#digest
#deep_learning
07.04.2019 15:04
https://pytorch.org/docs/master/nn.html#torch.nn.SyncBatchNorm
07.04.2019 15:04
Finally! Cool features like SyncBN or CyclicLR migrate to Pytorch!
31.03.2019 15:03
Miniaturize / optimize your ... NLP models?
For CV applications there literally dozens of ways to make your models smaller.
And yeah, I do not mean some "moonshots" or special limited libraries (matrix decompositions, some custom pruning, etc etc).
I mean cheap and dirty hacks, that work in 95% of cases regardless of your stack / device / framework:
- Smaller images (x3-x4 easy);
- FP16 inference (30-40% maybe);
- Knowledge distillation into smaller networks (x3-x10);
- Nave cascade optimizations (feed only Nth frame using some heuristic);
But what can you do with NLP networks?
Turns out not much.
But here are my ideas:
- Use a simpler model - embedding bag + plain self-attention + LSTM can solve 90% of tasks;
- Decrease embedding size from 300 to 50 (or maybe even more). Tried and tested, works like a charm. For harder tasks you lose just 1-3pp of your target metric, for smaller tasks - it is just the same;
- FP16 inference is supported in PyTorch for nn.Embedding, but not for nn.EmbeddingBag. But you get the idea;
_embedding_bag is not implemented for type torch.HalfTensor
- You can try distilling your vocabulary / embedding-bag model into a char level model. If it works, you can trade model size vs. inference time;
- If you have very long sentences or large batches - try distilling / swapping your recurrent network with a CNN / TCN. This way you can also trade model size vs. inference time but probably in a different direction;
#nlp
#deep_learning
26.03.2019 18:03
Dockerfile
d my DL/ML dockerfile with
- cuda 10
- PyTorch 1.0
https://github.com/snakers4/gpu-box-setup/
TF now also works with cuda 10
#deep_learning
26.03.2019 07:03
Russian sentiment dataset
In a typical Russian fashion - one of these datasets was d by the request of bad people, whom I shall not name.
Luckily, some anonymous backed the dataset up.
Anyway - use it.
Yeah, it is small. But it is free, so whatever.
#nlp
#data_science
25.03.2019 08:03
2019 DS / ML digest number 7
Highlights of the week
- NN normalization techniques (not batch norm);
- Jetson nano for US$99 released;
- A bitter lesson in AI;
https://spark-in.me/post/2019_ds_ml_digest_07
#digest
#deep_learning
25.03.2019 08:03
Good old OLS regression
I needed some quick boilerplate to create an OLS regression with confidence intervals for a very plain task.
Found some nice statsmodels examples here:
http://www.statsmodels.org/devel/examples/notebooks/generated/ols.html
#data_science