Here’s a big money quote from DYNOMIGHT in an assessment of cost to achieve AI scale for better intelligence (reduced loss):

…everyone reports that filtering the raw internet makes models better. They also report that including small but high-quality sources makes things better. But how much better? And why? As far as I can tell, there is no general “theory” for this. We might discover that counting tokens only takes you so far, and 10 years from now there is an enormous infrastructure for curating and cleaning data from hundreds of sources and people look back on our current fixation on the number of tokens with amusement.

10 years from now? There’s already a standard in infrastructure curating and cleaning data from hundreds of millions of sources: the W3C’s solidproject.org

Everyone says filtering data is better for intelligence? That’s nice to hear.

Nobody should believe in the absolute freedom of speech, when they factor high cost of predictable errors caused from data negligence.

How high? VERY high.

…the best current models have a total error of around 0.24 and cost around $2.5 million. To drop that to a total error of 0.12 would “only” cost around $230 million. …to scale a LLM to maximum performance would cost much more—with current technology, more than the GDP of the entire planet.

That 100X cost to reduce error seems prohibitive, although I’m sure someone is thinking $230 million is NBD like just one oligarch yacht.

DYNOMIGHT seems to be warning us current LLM architectures are about to run into a serious scale limit; they train already on all practically-available data with compute costs far too high.

Add integrity, gain intelligence at far lower cost to completely change the game. Serious food for thought considering Microsoft’s oligarchical ChatGPT has just been beaten by a pedestrian Web browser.

A month ago I asked Could you train a ChatGPT-beating model for $85,000 and run it in a browser?. $85,000 was a hypothetical training cost for LLaMA 7B plus Stanford Alpaca. “Run it in a browser” was based on the fact that Web Stable Diffusion runs a 1.9GB Stable Diffusion model in a browser, so maybe it’s not such a big leap to run a small Large Language Model there as well.

That second part has now happened.

Details are missing still in reports from Alabama. So far signs indicate yet another Tesla crashed at very high speed into a giant semi trailer.

Location: I-20/59 and Skyland Blvd East.
Time: 0150
Date: Today, April 15, Saturday
Deaths: Front two Tesla seats

The Tesla was pinned underneath the truck’s trailer and the vehicle’s four occupants were trapped, police said.

Pinned underneath sounds again like what we’ve seen with Tesla hitting trailers both from behind and the side. The fact it’s an intersection makes it sound like broadside, although it also could have been another case of exiting at high speed and hitting the rear of a parked trailer.

Local news hasn’t yet discussed whether the Tesla teens naively trusted the unsafe autopilot.

At least 17 people have been killed by Tesla’s known faulty automation software since June 2021 (when U.S. government required carmakers to report such failures). Hundreds more have died from the many other Tesla design and engineering defects.

Teslas reported driving under a trailer:

• 2016 Utah
• 2016 Florida
• 2019 Florida
• 2021 Michigan
• 2021 New Jersey
• 2022 Florida
• 2023 California

The petition makes a lot of sense. If you read between the lines, a transit safety expert proves again that Tesla really doesn’t know the basics about how to make a safe car.

In the petition, Lakafossis points out that it has been known since the 1980s that the Brake Transmission Shift Interlock (BTSI) feature reduces instances of sudden unintended acceleration in vehicles with automatic transmissions. The feature has also been required in new vehicles since 2010. “After proactively solving the problem of SUA accidents in the 1980s, today it seems that we are moving in the opposite direction, allowing a driver to select reverse gear while still driving forward without asking for a brake pedal application(!), actively encouraging drivers to enjoy ‘feet-off automatic braking and parking’,” Lakafossis says.

He has asked the National Highway Traffic Safety Administration (NHTSA) to recall all 1.6 million Tesla. Consideration of the ability to shift from drive to reverse without touching the brake pedal now sits along the Tesla recalls for steering wheels falling off, brakes not working, and seatbelt failures.

Has there ever been a worse car?