I have no data yet. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
| osintbuddy-plugins | osintbuddy.com | osintbuddy discord | osintbuddy-core-plugins |
Welcome to the OSINTBuddy project where you can connect, combine, and get insight from unstructured and public data as results that can be explored step-by-step. An easy-to-use plugin system allows any Python developer to quickly integrate new data sources so you can focus on discovering, interacting, and visualizing what's important to you
🚧 Work in progress 🚧
2023-12-12_19-07.webm
To get started please see CONTRIBUTING.md, and CONTRIBUTOR_AGREEMENT.md to agree with the AGPLv3
Table of Contents
What is osintbuddy.com?
Not much, yet.
But here's the concept:
An almost incomprehensible amount of data is created every day. And each year, figures are growing at an ever-increasing rate. These data sources can be divided up into six different categories of information flow:
- Public government data
- Media (newspapers, magazines, radio)
- Internet (blogs, discussion groups, citizen media, etc)
- Professional and academic publications (budgets, hearings, telephone directories, websites, etc)
- Commercial data (commercial imagery, financial assessments, databases, etc)
- Grey literature (technical reports, preprints, patents, business documents, etc)
OSINT is a method of working with, assessing and ranking information — We are living in the information age and the volume of information OSINT research has to deal with (information explosion) and the issue of low quality data leaves researchers drowning. The majority of this data is unstructured introducing many challenges to analyzing it and producing actionable intelligence since most data analytics databases are designed for structured data. We are surrounded by information and nearly none of it is useful.
We've decided to do something about it. The rapid developments in technologies such as AI and big data analytics have opened new horizons for OSINT which weren't previously available. We want to put all of that information at your fingertips. We want actionable intelligence quickly and to the point, that's why we're building an open-source OSINT tool that's free. Free to use, free to modify, free to do with as you wish, and built with plain old web technologies anyone can learn. But this isn't really the project. This is a free new community. A community for OSINT enthusiasts around the world and we need your help to design it, to program it, and to build it. We want to hear your suggestions, your ideas, and we're going to build it right in front of your eyes. The notion of a “needle in a haystack” is taken to the extreme on the internet. Let's build a magnet.
- Visual representation and layout modes for fetched data for easy understanding and editing.
- Simplified data fetching/transformations from custom sources using Python plugins, custom plugin/entity field layouts, check out the osintbuddy PyPi package for more details and expect many more input fields like files and checkboxes/toggles to come in the future.
- A development platform that is open for contributions.
- Works most of the time :)
At the core of this project lies a far more ambitious vision than the mere creation of an all-in-one data aggregation and analysis tool. What we aspire to start is the birth of a knowledge-driven community, passionately dedicated to the development of an evolving intelligence tool - a system and platform designed for discovering, interacting, and visualizing information to derive actionable insights. We imagine a system that not only learns from and evolves with your skills but also augments them. Here are some of our ideas for the future in no particular order:
- We want you to be able to interact with our algorithms, filter, segment, search for particular data, apply advanced query filters, data layouts, build queries visually, and more so you can uncover insights that matter via an easy-to-navigate web-based interface.
- We want to create a platform that promotes collective intelligence. Think of the currently popular collaborative workspaces such as AFFiNE, Logseq, or Obsidian but built into our data workspace. A system where each individual user contributes to the pool of knowledge, where you can share your graph environments, where you can track changes with annotations and comments, where insights from your workspaces can be optionally published and engaged with by a global community. Where every connection made, every data source integrated, every plugin created and shared by members of our community contributes to advancing insights around the world.
- We want functionality that can provide real-time updates for a set of entities or scans, notifying you when theres new data, changes, and or emerging patterns in your field of interest. Automatically retrieve insights as time goes on and learn from history with snapshots of your past data.
- We want to create a browser extension that allows you to seamlessly browse the web while also populating your OSINTBuddy graph. Each website you visit is mapped as an entity, with lines connecting them in the sequence you’ve traveled. Not just a bookmark list or a history tab, the history graph could maintain the context, showing not just where you’ve been, but also how and when you got there, and with the ability to extract a comment or piece of data out of a website you visit into an entity, you'll rarely have to leave your favorite web browser.
- Integrating AI, swarm intelligence, and evolutionary algorithms could let us create a tool that can continually evolve and optimize your operations over time. From NLP and sentiment analysis to search and anomaly detection with swarm intelligence, bringing data science tools and techniques to OSINTBuddy could let use create a powerful tool that reveals deep insights among a "haystack" of noisy information.
This is a project that will most likely not be finished for many years if ever, and that will require collaboration among experts in many fields. We're always looking for help, from writing documentation, researching feature ideas, designing the UX/UI, donating, to simply marketing and sharing the project, anything you contribute helps realize a vision for what could turn into a cross-disciplinary toolkit for working with information. Will you join me?
To start using OSINTBuddy, follow these simple installation steps:
Note that if you're on windows and want this project to work you need unix line endings (context). Before cloning, run: git config --global core.autocrlf false
Note that if you're running on an Apple device you will need to open your Docker app, select the features in development tab on the left hand side of the docker app, and enable/checkmark the Use Rosetta for x86/64 emulation on Apple Silicon option if you want this application to work
-
Clone the repo and submodules
git clone --recurse-submodules https://github.com/jerlendds/osintbuddy.git cd osintbuddy # using ssh? # git clone --recurse-submodules [email protected]:jerlendds/osintbuddy.git
-
Install Docker & Compose
-
Start the stack with Docker:
# ./launcher # display usage information. # ./launcher bootstrap # first time developing osintbuddy? read .github/CONTRIBUTING.md then run # bootstrap to setup the development environments for the stack. ./launcher start # start the osintbuddy app. # ./launcher stop # stop the osintbuddy app.
- Note: the stack will take a few minutes to startup while Solr and ScyllaDB configure themselves for JanusGraph. If you try to connect before all the databases are ready you will encounter errors.
- URLs
- Frontend: http://localhost:3000
- Casdoor: http://localhost:45910
- Backend: http://localhost:48997/api
- Documentation: http://localhost:48997/docs
- Access OSINTBuddy through the URLs provided for the frontend, backend, and documentation.
- Default login:
- username: osintbuddy
- password: osintbuddy
- Default login:
↑Roadmap
See the trello board for a list of our upcoming features.
See the forum: forum.osintbuddy.com, for a list of our upcoming features, to contribute feedback, ideas, and more (Note: We're currently in the process of theming and setting up the forum so we can move trello issues over. We're not sure when this will be completed. For the time being you can count on dev log updates being on the forum or on discord from now on).
See the open issues for a list of requested features (and known issues).
↑Contributing
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated. But if you'd like to make a significant change to this project or the osintbuddy-plugins project, please first create an issue or open a contributor post on the forum to get feedback before spending too much time. We don't want you to invest your time on changes we are already working on. Also, for details on how to get up and running with the project you can check out CONTRIBUTING.md. If you want to contribute directly please ensure you agree with the CONTRIBUTOR_AGREEMENT.md, in short, we want to ensure you're okay with your changes being licensed under the AGPLv3.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/add-core-feature) - Commit your Changes (
git commit -m 'feat: add core feature to osintbuddy app') - Push to the Branch (
git push origin feature/add-core-feature) - Open a Pull Request
Note: If you're working on the develop branch you want to ensure the osintbuddy-plugins submodule is also on the develop branch.
↑License
We are using the GNU Affero General Public License v3.0
In the early eighties, RMS realised that software was transformed from "a way to use a machine" to a product or a commodity. He foresaw that this would put an end to collective intelligence and to knowledge sharing. For the last 30 years, businesses and proponents of Open Source, including Linus Torvalds, have been decrying the GPL because of the essential right of "doing business" aka "privatising the common". Microsoft, through Github, Google and Apple pushed for MIT/BSD licensed software as the open source standard.
Our freedoms were taken away by proprietary software which is mostly coded by ourselves. For free. We spent our free time developing, debugging, testing software before handing them to corporations... Without Non-copyleft Open Source, there would be no proprietary MacOS, OSX nor Android. There would be no Facebook, no Amazon. We created all the components.
- The right to use the software at your own discretion
- The right to study the software
- The right to modify the software
- The right to redistribute the software, including with modifications
- The obligation to keep those four rights, effectively keeping the software in the commons.
We need to realize that any software without that last obligation will, sooner or later, become an oppression tool against ourselves. And that maintaining the commons is not only about software. It’s about everything we are as a society and everything we are losing against individual greed.
Note: the OSINTBuddy PyPi package is MIT licensed. We understand some individuals and businesses may not want to share their plugins developed in-house.
Patched aiogremlin library: jerlendds/gremlinpy
↑Contact
Open an issue if you need to get in touch with me or send an email to [email protected].
↑Sponsor OSINTBuddy
Help us keep the project free and maintained. Sponsor the OSINTBuddy project for unique benefits. Learn more on the OSINTBuddy OpenCollective