pinboard October 9, 2019

  • Climate change on Cape Cod: At the edge of a warming world – The Boston Globe
  • Twitter
    RT @mlcalderone: “This is a precedent that will destroy the concept of free and fair elections,” @jaketapper tells me. “It’s not rea…
  • Sea ‘Boiling’ with Methane Discovered in Siberia: ‘No One Has Ever Recorded Anything like This Before’
    Sea ‘Boiling’ with Methane Discovered in Siberia: ‘No One Has Ever Recorded Anything like This Before’
  • Twitter
    RT @laurahelmuth: For the first time on record, the 400 wealthiest Americans last year paid a lower total tax rate than any other inc…
  • Twitter
    RT @platonic:
  • EXCLUSIVE: The 2019 National Book Awards Finalists | Vanity Fair
  • Twitter
    RT @DigitalLawyer: Oooof. Was just subjected to the most credible phishing attempt I’ve experienced to date. Here were the steps:

    1)…

  • Twitter Took Phone Numbers for Security and Used Them for Advertising
  • [1909.12744] On the use of BERT for Neural Machine Translation
    Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently. In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation. We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality. We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments. In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness.
  • BERT model for Machine Translation · Issue #31 · huggingface/transformers
  • [1901.07291] Cross-lingual Language Model Pretraining
    BERT-related?
    Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT’16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT’16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.
  • [1902.04094] BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
    We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

Digest powered by RSS Digest

0 comments ↓

There are no comments yet...Kick things off by filling out the form below.

You must log in to post a comment.