First commit

.gitignore

@@ -0,0 +1,7 @@
+.idea
+.vscode
+**/*.rs.bk
+/*.data
+/*.gz
+*.local
+/target

CODE_OF_CONDUCT.md

@@ -0,0 +1,62 @@
+# Code of Conduct
+
+## 1. Purpose
+
+A primary goal of `crawly` is to be inclusive to the largest number of contributors, with the most varied and diverse backgrounds possible. As such, we are committed to providing a friendly, safe, and welcoming environment for all, regardless of gender, sexual orientation, ability, ethnicity, socioeconomic status, and religion (or lack thereof).
+
+This Code of Conduct outlines our expectations for all those who participate in our community, as well as the consequences for unacceptable behavior.
+
+## 2. Open Source Citizenship
+
+A supplemental goal of this Code of Conduct is to increase open source citizenship by encouraging participants to recognize and strengthen the relationships between our actions and their effects on our community.
+
+Communities mirror the societies in which they exist and positive action is essential to counteract the many forms of inequality and abuses of power that exist in society.
+
+## 3. Expected Behavior
+
+The following behaviors are expected and requested of all community members:
+
+- Participate in an authentic and active way.
+- Exercise consideration and respect in your speech and actions.
+- Attempt collaboration before conflict.
+- Refrain from demeaning, discriminatory, or harassing behavior and speech.
+- Be mindful of your surroundings and of your fellow participants.
+
+## 4. Unacceptable Behavior
+
+The following behaviors are considered harassment and are unacceptable within our community:
+
+- Violence, threats of violence or violent language directed against another person.
+- Sexist, racist, homophobic, transphobic, ableist, or otherwise discriminatory jokes and language.
+- Posting or displaying sexually explicit or violent material.
+- Posting or threatening to post other people’s personally identifying information ("doxing").
+- Personal insults, particularly those related to gender, sexual orientation, race, religion, or disability.
+- Inappropriate photography or recording.
+- Unwelcome sexual attention.
+- Advocating for, or encouraging, any of the above behavior.
+
+## 5. Consequences of Unacceptable Behavior
+
+Unacceptable behavior from any community member, including sponsors and those with decision-making authority, will not be tolerated. Anyone asked to stop unacceptable behavior is expected to comply immediately.
+
+If a community member engages in unacceptable behavior, the community organizers may take any action they deem appropriate, up to and including a temporary ban or permanent expulsion from the community without warning.
+
+## 6. Reporting Guidelines
+
+If you are subject to or witness unacceptable behavior, or have any other concerns, please notify a community organizer as soon as possible.
+
+## 7. Addressing Grievances
+
+Only permanent resolutions (such as bans) may be appealed. To appeal a decision of the working group, contact the project team at [[email protected]](mailto:[email protected]) with your appeal and the working group will review the decision.
+
+## 8. Scope
+
+We expect all community participants (contributors, paid or otherwise; sponsors; and other guests) to abide by this Code of Conduct in all community venues–online and in-person–as well as in all one-on-one communications pertaining to community business.
+
+## 9. Contact info
+
+[[email protected]](mailto:[email protected])
+
+## 10. License and attribution
+
+This Code of Conduct is distributed under a Creative Commons Attribution-ShareAlike license. Portions of the text are derived from the Django Code of Conduct and the Geek Feminism Anti-Harassment Policy.

CONTRIBUTING.md

@@ -0,0 +1,71 @@
+# Contributing to crawly 🕷️
+
+First off, thank you for considering contributing to `crawly`! It's people like you that make `crawly` such a great tool.
+
+## Code of Conduct
+
+Every community member is expected to follow our [Code of Conduct](CODE_OF_CONDUCT.md). Please make sure you are welcoming and respectful to everyone.
+
+## Where can I ask for help?
+
+If you have any questions, feel free to ask in the [issues section](https://github.com/CrystalSoft/crawly/issues). Label your issue as `question`.
+
+## How to contribute
+
+1. **Fork the repository**: Start by forking the `crawly` repository.
+
+2. **Clone the forked repository**:
+
+```bash
+git clone https://github.com/YOUR-USERNAME/crawly.git
+cd crawly
+```
+
+3. **Create a new branch**:
+```bash
+git checkout -b my-new-feature
+Make Changes: Add your feature or fix bugs and make sure to commit your changes.
+```
+
+4. **Push to GitHub**:
+```bash
+git push origin my-new-feature
+```
+
+5. **Submit a Pull Request**: Go to the crawly GitHub page and click on "New Pull Request". Select your fork and the branch you created. Click "Create Pull Request".
+
+6. **Address Feedback**: Sometimes, your changes might not be accepted as is. Typically, a maintainer will review and ask for any necessary changes before they merge it.
+
+## 🐛 Reporting Bugs
+
+1. **Check Existing Issues**: Ensure that the bug was not already reported by searching on GitHub under [Issues](https://github.com/CrystalSoft/crawly/issues).
+
+2. **Provide Information**: If the bug hasn't been reported, open a new issue. Fill out the bug report with details like:
+    - **Description of the problem**: A clear and concise description of what the bug is.
+    - **Steps to reproduce**: Provide step-by-step sequence to reproduce the issue.
+    - **Expected behavior**: Describe what you expected to happen.
+    - **Actual behavior**: What actually happened.
+    - **Screenshots**: If applicable, add screenshots to help explain the problem.
+    - **Environment details**: Mention your OS, Rust version, and any other relevant details.
+
+## ✨ Suggesting Enhancements
+
+1. **Check Existing Issues**: Ensure that the enhancement hasn't been suggested by searching the [issues list](https://github.com/CrystalSoft/crawly/issues).
+
+2. **Provide Information**: If you have a new idea, open a new issue. Detail the enhancement with:
+    - **A clear and descriptive title**: This helps others understand the suggestion.
+    - **Detailed description**: Explain the core ideas of the enhancement.
+    - **Reasons**: Why do you believe this enhancement would be beneficial.
+    - **Use case scenarios**: Describe how this enhancement might be used by end-users.
+
+## 🛠️ Setting up your development environment
+
+1. **Install Rust**: If you haven't already, install Rust by following the instructions [here](https://www.rust-lang.org/learn/get-started).
+
+2. **Fork and clone**: Follow the steps mentioned in the "How to contribute" section.
+
+3. **Build**: Run `cargo build` to ensure everything is set up correctly.
+
+4. **Run tests**: Before pushing any changes, ensure you run and pass all tests with `cargo test`.
+
+Remember, contributing isn't just about code! You can help by improving documentation, designing, and more. Every contribution is appreciated.