Open Source AI is open-washing by any way of looking at it [LWN.net]

Open Source AI is open-washing by any way of looking at it

Posted Nov 5, 2024 12:34 UTC (Tue) by lmb (subscriber, #39048)
In reply to: Open Source AI is open-washing by any way of looking at it by zack
Parent article: OSI board AMA at All Things Open

I don't want to equate whether opening the data up is good or bad directly either. I, too, have been involved in this topic for three decades, I realize the truth is rarely pure and never simple.

There can be good reasons for data not to be made public or released openly. Medical, safety, personal privacy - and yet the AI/ML systems trained on them serve vital functions for society. Those may only be visible to officially appointed & chartered inspectors, for example.

Even the FSF acknowledges this - nonfree systems can still be just.

No, open/free/public data sets are not the only way to respect user rights.

Where the OSAID falls short in my opinion is insisting that those systems are "Open Source AI" - not everything can be open, and that's fine. (Or they could, indeed, say they're Open(tm) _only_ if they fall under such exempt regulations and are indeed independently reviewable.)

The *default* for an "Open Source AI" system should, in my book, indeed be open data. (And it'd be helpful if OSAID specified terms one could comply with.)

Again, my gripe is OSI going with such an all-encompassing term and indeed claiming their definition covers all components comprehensively - from a very prominent position with a lot of power and influence.

I think their OSAID 1.0 should have been more nuanced & differentiated and mostly stick to the parts we do understand reasonably well. And deliver something we can actually implement in practice. This, to me, reeks of preempting regulatory decisions, and/or marketing reasons.

I don't want in-fighting while the absolute exploitationists rejoice, either. But the OSI started the overreach with a reductionist definition, and also claims the high ground of authority - they get judged accordingly. They don't get free cheerleading.

The fact that so many assumed-to-be-well-meaning people see this as a potential erosion and open-washing clearly shows they've not produced something that is clear enough, if that truly isn't their intent.

(I also know you can't ever produce anything that is 100% proof against misinterpretation by malicious actors, but the folks whom I've seen voice criticism don't tend to fall into that camp.)

to post comments

Open Source AI is open-washing by any way of looking at it

Posted Nov 5, 2024 13:38 UTC (Tue) by paulj (subscriber, #341) [Link]

I think that for any generally queryable LLM made available to the public, you must be willing to accept the user is able to access (at least) snippets of the original training data. That is, the sensitivity of access to the prompt will be equal to the sensitivity of access to the original data. You need to treat the two - prompt and training data - as equivalent in terms of access.

In practical terms, for such LLMs, I think it will be required to anonymise any sensitive data.

It could be there are other kinds of AI models that can not, of themselves, directly leak the input data. E.g., a model arranged and trained to classify, say, diseases. You chat to it with your symptoms, perhaps, and it spits out a disease, and only a disease. If the output layers can only a set of output symbols that is a distinct set from the training data and much more limited than the training data, you could argue it is "safe" to distribute that model.

However, still, the training data is encoded into the parameters. The model state is a compressed form of the input. For sensitive input data, you would /not/ want to bet that it is safe to distribute the parameters, just cause they end up selecting from a predetermined and limit output layer. You would /not/ want to bet that some clever AI-hacker will eventually figure out how to backward-engineer some (or more) of the training data from the parameters.

Open Source AI is open-washing by any way of looking at it

Posted Nov 5, 2024 13:51 UTC (Tue) by zack (subscriber, #7062) [Link]

I completely agree with you on nuance.

I've also participated, as a volunteer, in the OSAID process. And once it became clear that OSI was opposed to mandating training data access, I've "battled" (for lack of a better term) for either a two-term definition (e.g., "open weight" vs "open source") or an additional qualifier (e.g., level 1/2/3/4), depending on whether training data were open data/public data/obtainable data/private data. I regret having lost that battle too. (But I still think that OSAID is better than nothing, given the current state of the AI industry, and that it will play a positive role in upcoming regulations.)