Страница телеграм канала Spark in me - Internet, data science, math, deep learning, philosophy
Spark in me - Internet, data science, math, deep learning, philosophy
1332 подписчиков
All this - lost like tears in rain. Internet, data science, math, deep learning, philosophy. No bs. Our website - http://spark-in.me Our chat - https://goo.gl/WRm93d DS courses review - http://goo.gl/5VGU5A - https://goo.gl/YzVUKf
Входит в категории: Технологии
31.03.2020 16:03
Towards End-to-end ASR
Towards End-to-end ASR - an internal (?) presentation by Google
https://drive.google.com/file/d/1Rpob1-C223L9UWTiLJ6_Dy12mTA3YyTn/view
This is such a huge corpus of work.
Interesting conclusions:
- Google writes your voice (at least from Google assistant, unclear whether they abuse their "phone" app) and uses this data for their models. Surprise surprise!
- Obviously Google is pushing towards end-to-end ASR within one NN on a mobile device for a number of reasons:
(i) easier packaging
(ii) quantization
(iii) no requirement to run a large LM alongside the model
(iv) Google has a lot of data (end-to-end models suffer from lack of data mostly)
- 120MB total system size on mobile device. This means AM + LM, which in this case is one quantized RNN-T model (4x - float32 => int8)
- They also write that hybrid systems with LM fusion / rescoring perform better overall
- The "best" cited solutions are not end-to-end, though
- Finally understood why they were pushing their RNN-T models instead of 10x more frugal alternatives. Old and optimized layers, hacks to speed up inference, unlimited resources, better performance (on the same step). Also LSTMs are known to be able to replace LMs
- Google also knows about "Time Reduction Layer", but looks like when using it within and RNN it is a bit painful - a lot of fiddling in the model logic
- Looks like given unlimited resources, data and compute - the easiest solution is to train large LSTMs in an end-to-end fashion (I also noticed that LSTMs have higher quality on same step, but MUCH weaker speed and convergence overall in terms of time-to-accuracy), optimize it heavily, quantize and deploy
- Sharing AMs / LMs for different dialects kind of works (maybe in terms of time-to-accuracy?), but direct tuning is better
But is full 100% end-to-end feasible on any scale below Google?
Probably not. Unless you are Facebook.
Having a fully end-2-end pipeline will have OOV (even with BPE / word-piece tokens) and other problems - like bias towards domains where you have audio. It will certainly NOT generalize towards unseen new words and pronunciations.
Meh.
But can you have extremely small mobile models?
Yes and no. Our latest small AM is targeting 200MB before quantization and probably 50MB after. Current production model is around 90MB (after quantization). But can it serve instead of an LM?
Technically yes, but quality will suffer. Unlike Google we do not have unlimited data, compute and low level engineers. On the other hand fully neural post-processing / decoding w/o huge transformer-like models is more than feasible. So we will see =)
#speech
30.03.2020 12:03
2020 DS / ML Digest 4
Highlights
- Google tries to ... reduce compute used for transformer pre-training
- Some NLP datasets - Russian news and Google dialogues
- Very cool story of a failed autonomous truck company
- The state of autonomous vehicles
Please like / share / repost!
https://spark-in.me/post/2020_ds_ml_digest_04
#digest
28.03.2020 22:03
Towards an ImageNet Moment for Speech-to-Text
First CV, and then (arguably) NLP, have had their ImageNet moment — a technical shift that makes tackling many problems much easier. Could Speech-To-Text be next?
Following the release of our production models / metrics, this is our piece on this topic on thegradient.pub! So far this is the largest work ever we have done, and I hope that it will not go under the radar.
It is in our hands now to make sure that speech recognition brings value to people worldwide, and not only some fat cats.
So, without further ado:
- The piece itself https://thegradient.pub/towards-an-imagenet-moment-for-speech-to-text/
- Some more links here https://spark-in.me/post/towards-an-imagenet-moment-for-speech-to-text
- If you are on Twitter, please repost this message - https://twitter.com/gradientpub/status/1243967773635571712
A lot of thanks to Thegradient team, especially Andrey and Jacob, for the sheer amount of work they put in to make this piece readable and understandable!
Please like, share, repost!
Also, there will be a second piece with criticism, so stay tuned!
#speech
#deep_learning
28.03.2020 07:03
Surprisingly this did not work.
My guess - because of how CTC networks learn blanks and the signal is dilluted.
Just tuning the hyper-params gave me 40% further weight reduction, but the networks are still ~2pp CER off.
Maybe it will work with networks trained wo CTC?
26.03.2020 09:03
Russian Speech Recognition
You may have heard about our dataset, Open STT.
And yeah, if you have not guessed we have built a Speech-To-Text system that is better / on par with alleged "market leaders", the only difference being that we publish something for the common good and we do not need 100 Tesla GPUs (wink-wink, Oleg).
Also if it is not super obvious, this thing is already deployed into production and it really works.
Now we decided to go out of stealth mode a bit and publish a series of publications in online Russian / English publications:
- A piece on Habr.com - just published https://habr.com/ru/post/494006/ - it is very short and abridged, you know habr;
- 2 more detailed pieces on https://thegradient.pub - coming soon!
If you want more gory details, you can see a couple of posts on our projects website:
- STT system quality benchmarks - https://www.silero.ai/russian-stt-benchmarks/
- STT system speed https://www.silero.ai/stt-system-speed/
- How to measure quality in STT https://www.silero.ai/stt-quality-metrics/
If you would like to test our system, you may first want to:
- Try a demo http://demo.silero.ai/ (more beautiful mobile demo coming up!)
- See the API https://api.silero.ai/docs
#deep_learning
#speech
#nlp