Страница телеграм канала Spark in me - Internet, data science, math, deep learning, philosophy
Spark in me - Internet, data science, math, deep learning, philosophy
1332 подписчиков
All this - lost like tears in rain. Internet, data science, math, deep learning, philosophy. No bs. Our website - http://spark-in.me Our chat - https://goo.gl/WRm93d DS courses review - http://goo.gl/5VGU5A - https://goo.gl/YzVUKf
Входит в категории: Технологии
09.11.2021 11:11
INT8 Optimizer + PyTorch Native AMP
So, we tried the recent INT8 optimizer, also together with PyTorch native AMP.
We were also migrating this job from 1080 Ti to 3090 based environment.
Maybe the network is too small (~100M params including the embedding layer), but looks like apples to apples 3090 provides a 3x throughput boost, and AMP + INT 10-15% boost tops.
Well, I understand the selling point of being able to tune large networks, but ultimately on real down-to-earth task the speed improvement is marginal.
It is very cool that this even works and the optimizations described look very cool and sane, but the political underpinnings look very much like marginalization of independent research. I.e. if you cannot afford 1000 GPUs, then please do not even try anything or just enjoy scraps from our table.
#deep_learing
05.11.2021 17:11
Бесплатное распознавание речи для всех желающих
https://habr.com/ru/post/587512/
02.11.2021 10:11
Similar results from MS
No mention of exact compute and / or resources required, but:
- 2.5-billion parameter Universal Image Language Representation model - 94 languages
What impresses me though is that contrastive learning on this scale enables making a model really multi-lingual, because images obviously transcend language.
All of this is not accessible to people ofc, but this is one of rare occasions where "large models" can be actually useful.
https://www.microsoft.com/en-us/research/blog/turing-bletchley-a-universal-image-language-representation-model-by-microsoft/
01.11.2021 09:11
Digest 2021-10
Datasets
The Face Synthetics dataset
- https://github.com/microsoft/FaceSynthetics
- 100,000 images of faces at 512 x 512 pixel resolution
- 70 standard facial landmark annotations
- per-pixel semantic class anotations
- 100,000 images (32GB)
Announcing WIT: A Wikipedia-Based Image-Text Dataset - https://ai.googleblog.com/2021/09/announcing-wit-wikipedia-based-image.html
- Images 11.5M Text 37.5M Contextual Text ~119M Languages 108
We started by selecting Wikipedia pages that have images, then extracted various image-text associations and surrounding contexts. To further refine the data, we performed a rigorous filtering process to ensure data quality. This included text-based filtering to ensure caption availability, length and quality (e.g., by removing generic default filler text); image-based filtering to ensure each image is a certain size with permissible licensing; and finally, image-and-text-entity–based filtering to ensure suitability for research (e.g., excluding those classified as hate speech). We further randomly sampled image-caption sets for evaluation by human editors, who overwhelmingly agreed that 98% of the samples had good image-caption alignment.
PASS: An ImageNet replacement for self-supervised pretraining without humans:
- https://arxiv.org/abs/2109.13228
- https://www.robots.ox.ac.uk/~vgg/research/pass/explorer/
- 1.28M Internet images with CC-BY license that do not contain humans or body parts at all
AP-10K: A Benchmark for Animal Pose Estimation in the Wild - https://github.com/AlexTheBad/AP-10K
The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.
- https://pile.eleuther.ai/
GoEmotions: A Dataset for Fine-Grained Emotion Classification:
- https://ai.googleblog.com/2021/10/goemotions-dataset-for-fine-grained.html
- a human-annotated dataset of 58k Reddit comments extracted from popular English-language subreddits and labeled with 27 emotion categories
#digest
01.11.2021 09:11
Digest 2021-10
Tech
Q&A on Mental Health – Part 1 (minus bs books) - https://unstructed.tech/2021/09/29/qa-on-mental-health-part-1/
Q&A on Mental Health – Part 2 - https://unstructed.tech/2021/10/07/qa-on-mental-health-part-2/
Similar from Durov - https://t.me/durov/171
CHINA’S PATENT MARCH - https://digitstodollars.com/2021/10/05/chinas-patent-march/
PATENTS ARE THE NEW OIL? - https://digitstodollars.com/2021/10/06/patents-are-the-new-oil/
About new HTTP2+ protocols - https://aminux.wordpress.com/2021/10/09/quic-and-fancy-protocols-vs-shit-websites/
GLOBAL FOUNDRIES IPO!!! - https://digitstodollars.com/2021/10/08/global-foundries-ipo/
Blogs
Stepping out of the firehose - https://www.ben-evans.com/benedictevans/2021/10/10/stepping-out-of-the-firehose
The death of the newsfeed - https://www.ben-evans.com/benedictevans/2018/4/2/the-death-of-the-newsfeed
Metaverse! Metaverse? Metaverse!! - https://www.ben-evans.com/benedictevans/2021/10/9/metaverse-metaverse-metaverse
Looking at the outlying data points - https://rachelbythebay.com/w/2021/10/13/temp/
Willingness to look stupid - https://danluu.com/look-stupid/
CONTENT IS EATING INFRASTRUCTURE - https://digitstodollars.com/2021/10/20/content-is-eating-infrastructure/
#digest
01.11.2021 09:11
Digest 2021-10
Code
Whats new in Python 3.11 - https://docs.python.org/3.11/whatsnew/3.11.html
Why you can’t switch to Python 3.10 just yet - https://pythonspeed.com/articles/switch-python-3.10/
Conda vs pip - https://pythonspeed.com/articles/conda-vs-pip/
Faster Docker builds with pipenv, poetry, or pip-tools - https://pythonspeed.com/articles/pipenv-docker/
OSS contribution reality - https://habr.com/ru/post/583782/
Secure Password Handling in Python - https://martinheinz.dev/blog/59
Взлом ГПСЧ с помощью машинного обучения - https://habr.com/ru/company/dcmiran/blog/584692/
Match/case in python 3.10 - https://habr.com/ru/post/585216/#comment_23627514
What’s running in production? Making your Docker images identifiable - https://pythonspeed.com/articles/identifying-images/
#digest
01.11.2021 09:11
Digest 2021-10
ML / Papers
Introducing Omnimattes: A New Approach to Matte Generation using Layered Neural Rendering
- https://ai.googleblog.com/2021/08/introducing-omnimattes-new-approach-to.html
- A new rendering network is trained for each video
"Hello, Its Me": Deep Learning-based Speech Synthesis Attacks in the Real World - http://arxiv.org/abs/2109.09598
Introducing FLAN: More generalizable Language Models with Instruction Fine-Tuning - https://ai.googleblog.com/2021/10/introducing-flan-more-generalizable.html
PICO: Pragmatic Compression for Human-in-the-Loop Decision-Making - https://bair.berkeley.edu/blog/2021/10/06/pico/
Building Scalable, Explainable, and Adaptive NLP Models with Retrieval - http://ai.stanford.edu/blog/retrieval-based-NLP/
Paper Highlight: GPT-3 Summarizes Books - https://thegradientpub.substack.com/p/gradient--10-gpt-3-summarizes
DEEP LEARNING’S DIMINISHING RETURNS - https://spectrum.ieee.org/deep-learning-computational-cost
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World’s Largest and Most Powerful Generative Language Model - https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
Microsoft Translator: Now translating 100 languages and counting! - https://www.microsoft.com/en-us/research/blog/microsoft-translator-now-translating-100-languages-and-counting/
Baidu Releases PLATO-XL: World’s First 11 Billion Parameter Pre-Trained Dialogue Generation Model - http://research.baidu.com/Blog/index-view?id=163
Fake It Till You Make It - https://microsoft.github.io/FaceSynthetics/
Selective Classification Can Magnify Disparities Across Groups - http://ai.stanford.edu/blog/sc-magnifies-disparities/
A New Link to an Old Model Could Crack the Mystery of Deep Learning - https://www.quantamagazine.org/a-new-link-to-an-old-model-could-crack-the-mystery-of-deep-learning-20211011/
Deciding Which Tasks Should Train Together in Multi-Task Neural Networks - https://ai.googleblog.com/2021/10/deciding-which-tasks-should-train.html
Practical Differentially Private Clustering - https://ai.googleblog.com/2021/10/practical-differentially-private.html
Grammar Correction as You Type, on Pixel 6 - https://ai.googleblog.com/2021/10/grammar-correction-as-you-type-on-pixel.html
Bitsandbytes is a lightweight wrapper around CUDA custom functions, in particular 8-bit optimizers and quantization functions. - https://github.com/facebookresearch/bitsandbytes
How Underspecification Presents Challenges for Machine Learning - https://ai.googleblog.com/2021/10/how-underspecification-presents.html
The State of "AI":
- Warning - no critical analysis, just hype copypasta
- Email - http://newsletter.airstreet.com/issues/state-of-ai-report-2021-694504
- Blog post - https://www.stateof.ai/2021-report-launch.html
- Slides themselves - https://docs.google.com/presentation/d/1bwJDRC777rAf00Drthi9yT2c9b0MabWO5ZlksfvFzx8/edit#slide=id.gf171287819_0_165
DALL-E mini - how real is the OpenAI bs - https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-mini - Vmlldzo4NjIxODA
#digest
29.10.2021 09:10
8-bit Optimizers via Block-wise Quantization
While this is very cool in theory and on paper you allegedly reduce your optimizer memory by 50%+, this is very loaded politically and practically: - Most likely this would work well only on Ampere GPUs (though they list Requirements: anaconda, cudatoolkit, pytorch Hardware requirements: NVIDIA Maxwell GPU or newer (>=GTX 9XX) Supported CUDA versions: 9.2 - 11.3); - This is heavily loaded with perverse incentives. Huge LLMs are yet to prove the investment, but lets build even bigger models, i.e. trillion-param sized models!; - This is maintained by FAIR, but packaged separately. So it can go either way - be abandoned after being used in achieving new trillion param model record or be then somehow merged into PyTorch or even maybe Nvidia products; I can see one real practical use-case for this. If e.g. on older hardware you are training a large model (100 - 200M params) and you have problems with small batch-size, this can be a life-saver (or in case of Video or 3D or similar). Otherwise I am not sure. Also note, they do not mention speed or convergence speed on different GPUs. #deep_learing
While this is very cool in theory and on paper you allegedly reduce your optimizer memory by 50%+, this is very loaded politically and practically: - Most likely this would work well only on Ampere GPUs (though they list Requirements: anaconda, cudatoolkit, pytorch Hardware requirements: NVIDIA Maxwell GPU or newer (>=GTX 9XX) Supported CUDA versions: 9.2 - 11.3); - This is heavily loaded with perverse incentives. Huge LLMs are yet to prove the investment, but lets build even bigger models, i.e. trillion-param sized models!; - This is maintained by FAIR, but packaged separately. So it can go either way - be abandoned after being used in achieving new trillion param model record or be then somehow merged into PyTorch or even maybe Nvidia products; I can see one real practical use-case for this. If e.g. on older hardware you are training a large model (100 - 200M params) and you have problems with small batch-size, this can be a life-saver (or in case of Video or 3D or similar). Otherwise I am not sure. Also note, they do not mention speed or convergence speed on different GPUs. #deep_learing
25.10.2021 18:10
As an experiment reposted last article to VC.ru just for lulz:
- https://vc.ru/tribuna/310183-sinteziruem-golos-babushki-dedushki-i-lenina-novosti-nashego-publichnogo-sinteza
22.10.2021 10:10
Willingness to look stupid
Be stupid, think and experiment, do not take the bs. Just say what is on your mind.
It is so refreshing and inspiring to see such a blog post from an American, their culture being based on hypocrisy, back-stabbing and segregation.
https://danluu.com/look-stupid/
#no_bs
21.10.2021 19:10
PyTorch 1.10 Release, including CUDA Graphs APIs, Frontend and compiler improvements
PyTorch 1.10 s are focused on improving training and performance of PyTorch, and developer usability. The full release notes are available here. Highlights include:
- CUDA Graphs APIs are integrated to reduce CPU overheads for CUDA workloads.
- Several frontend APIs such as FX, torch.special, and nn.Module Parametrization, have moved from beta to stable.
- Support for automatic fusion in JIT Compiler expands to CPUs in addition to GPUs.
- Android NNAPI support is now available in beta.
https://github.com/pytorch/pytorch/releases/tag/v1.10.0
https://pytorch.org/blog/pytorch-1.10-released/