Project summary

With just a few images scraped from social media, someone can create a realistic image that didn’t originally exist of you. These AI-generated faces can then be used to impersonate you, harass you or otherwise be abused.

There are ways to protect photographs from being used to train AI models including watermarks and protection processes designed to confuse AI models. However, there’s limited public information about whether they can actually prevent their images from being abused if someone is motivated to overcome the protection.

I’ve worked professionally in offensive security for nearly a decade and a half. I’d like to use that background to test some protections and see if they actually live up to their claims and reduce the likelihood someone’s identity can be abused.

Goals & Approach

The primary objective of this research is to determine if watermarked or protected photos can still be used to create realistic deepfakes. We hope to test five or six different protection processes against four or five AI image generation models. These tests will include evasion attempts, repeated processing, and additional transformations typically seen as images are shared over the internet.

We will also run tests against existing deepfake detectors, face match technology, and provenance validation techniques to determine how often they fail to detect manipulation or flag legitimate images.

The research will run on approximately 1,000-2,000 licensed or consent obtained images. This will include a set of specially created synthetic images to use as identities.

The first 90 days will be dedicated to research and designing the initial tests. The second 90 days will be dedicated to larger scale testing. The final 90 days will be dedicated to analysis as well as peer review and drafting the paper for publication.

All testing will only be performed on images we have obtained explicit permission to use. We will not be scraping private photos of individuals or creating explicit images of real people.

Deliverables

Empirical evidence supporting which identity protection mechanisms actually help prevent identity abuse, in which situations, and how well they work.

How will you measure impact?

We will measure failed protections, false detections, as well as image quality alterations. Results will name and shame techniques that didn’t perform well.

The results will be published in enough detail that they can be repeated by other academics. We also intend to create a scientific publication, public handbook, and free curriculum content for use by 404 Founders Academy.

What are other relevant details about the project?

See below for our budget and information about how we’ll use funds if the project is fully funded.

How will funds be spent?

The ask is for $78,000 to cover 9 months of work. We’ve budgeted $24,000 for scientific analysis & image analysis, $21,600 for security analysis & technical assessment, and $18,000 to pay for research assistance. Additional funds include $5,000 for computing & research data, $4,000 for independent review & publication, and $5,400 for administration & legal.

Research will be coordinated through 404Founders OÜ. The company’s Scientific & Technical Advisory Board contains an academic professor who has published research on image processing and media manipulation detection. We can provide his academic CV to funders who are interested.

Our minimum request is $26,000. This would allow us to run a smaller 3-month study and publish initial findings publicly.

This project is separately budgeted from any of our other research projects or grant requests.