Blog

  • TestSSR

    TestSSR

    This project was inspired by SSAccount.py

    SSAccount.py provided some codes which can verify the availability of shadowsocks account.

    http://bindog.github.io/blog/2014/09/06/shadowsocks-account/
    http://pan.baidu.com/s/1mgr6XPu password: 2i9y

    At the beginning, I just revised some codes, use thoese codes to find out optimum accounts by sorting latencies.

    I wrote some python scripts to collect free accounts from http://doub.bid/sszhfx , https://github.com/Alvin9999/new-pac/wiki/ss%E5%85%8D%E8%B4%B9%E8%B4%A6%E5%8F%B7 and other websites. Those scripts generate some json files. I can execute another tiny python scripts to combine those json files into one json file. Everyday I can collect about 95-110 free accounts.

    Soon I found that latency was not good measurement for figuring out optimum accounts. I found a bandwidth speed test ultility, speedtest-cli. One day after I integrated speedtest-cli into my test program, I thought test youtube video download speed is better, instead of bandwidth speed test. I found youtube-dl in no time.

    At last, I tinkered with youtube-dl, subprocess, urllib2, requests and other python modules, created a test program for sorting out optimum accounts from dozens of free accounts.

    Pre-req:

    pip install youtube-dl

    pip install requests

    Visit original content creator repository
    https://github.com/ggzzzzzzz/TestSSR

  • awesome-python-security


    A curated list of awesome Python security related resources.

    Awesome

    List inspired by the awesome list thing.

    Supported by: GuardRails.io


    Contents

    Tools

    Web Framework Hardening

    • Secure.py – secure.py 🔒 is a lightweight package that adds optional security headers and cookie attributes for Python web frameworks.
    • Flask-HTTPAuth – Simple extension that provides Basic, Digest and Token HTTP authentication for Flask routes.
    • Flask Talisman – Talisman is a small Flask extension that handles setting HTTP headers that can help protect against a few common web application security issues.
    • Django deployment checklist – Web framework Django has built-in feature to check for security configurations: run this command manage.py check --deploy. It’s really helpful as it already included in the framework.
    • Django Session CSRF – CSRF protection for Django without cookies.

    Multi tools

    • hawkeye – Multi purpose security/vulnerability/risk scanning tool supporting Ruby, Node.js, Python, PHP and Java.
    • GuardRails – A GitHub App that gives you instant security feedback in your Pull Requests.
    • Hubble – Hubble is a modular, open-source security compliance framework.
    • Salus – Multi purpose security scanning tool supporting Ruby, Node, Python and Go.

    Static Code Analysis

    • Bandit – Bandit is a tool designed to find common security issues in Python code.
    • Pyt – A Static Analysis Tool for Detecting Security Vulnerabilities in Python Web Applications.
    • Detect Secrets – An enterprise friendly way of detecting and preventing secrets in code.

    Vulnerabilities and Security Advisories

    Penetration Testing

    • EvilTwinFramework – A framework for pentesters that facilitates evil twin attacks as well as exploiting other wifi vulnerabilities.
    • sqlmap – Automatic SQL injection and database takeover tool

    Cryptography

    • Passlib – Secure password storage/hashing library, very high level.
    • PyNacl – Python binding to the Networking and Cryptography (NaCl) library.

    Application Templates

    Educational

    Hacking Playground

    • Let’s be bad Guys – Shiny, Let’s Be Bad Guys: Exploiting and Mitigating the Top 10 Web App Vulnerabilities.
    • django.nV – django.nV is a purposefully vulnerable Django application provided by nVisium.
    • DSVW – Damn Small Vulnerable Web (DSVW) is a deliberately vulnerable web application written in under 100 lines of code, created for educational purposes.
    • DVPWA – Damn Vulnerable Python Web Application was inspired by famous dvwa project and bobby-tables xkcd comics.

    Books

    Articles, Guides & Talks

    • cryptography – A package designed to expose cryptographic primitives and recipes to Python developers.
    • 10 Common Security Gotchas in Python – 10 common security gotchas in Python and how to avoid them.
    • OWASP Python Security – Aims at creating a hardened version of python that makes it easier for developers to write applications more resilient to attacks and manipulations.
    • Django Security – Overview of Django’s security features includes advice on securing a Django-powered site.

    Companies

    • GuardRails – A GitHub App that gives you instant security feedback in your Pull Requests.
    • Snyk – A developer-first solution that automates finding & fixing known vulnerabilities in your dependencies.

    Other

    Reporting Bugs

    Contributing

    Found an awesome project, package, article, or another type of resources related to Python Security? Send me a pull request! Just follow the guidelines. Thank you!


    say hi on Twitter

    License

    CC0

    Visit original content creator repository https://github.com/guardrailsio/awesome-python-security
  • nextjs-example

    Next.js Example Template

    This repository provides a ready-made template for Next.js projects to help you quickly get started with a modern web application. It includes a basic project setup with the latest Next.js features and best practices.

    Features

    • Next.js: The latest version of Next.js to power your project.
    • Pre-configured ESLint: Ensures code quality with pre-configured ESLint rules.
    • Pre-commit & Pre-push: Automatically run lint checks and build processes before committing or pushing changes.
    • Tailwind and Import Sorter: Automatic sorting for Tailwind CSS classes and JavaScript/TypeScript imports.

    Getting Started

    To create a new project based on this template, run the following command:

    npx create-next-app my-nextjs-app -e https://github.com/Mr-Meshky/nextjs-example/

    This will create a new directory my-nextjs-app with all the files from this template.

    Running the Project

    Once you’ve created the project, navigate into the project directory:

    cd my-nextjs-app

    Install the necessary dependencies:

    pnpm install

    Start the development server:

    pnpm run dev

    Now open http://localhost:3000 to view the project in your browser.

    Customization

    Feel free to modify the template to suit your needs. You can edit any component, create new pages, or add additional libraries as needed.

    Visit original content creator repository
    https://github.com/Mr-Meshky/nextjs-example

  • din

    din

    An unofficial Dart library for Discord.

    Uses Dart 1.25.0-devLicense Pub Package Build Status

    GitHub Issues GitHub Forks GitHub Stars

    Install

    The current pre-release of din does not require other runtime dependencies.

    dependencies:
      din: ^0.1.0-alpha+5

    To get started, it’s recommended to read the Discord documentation and create an application for API access.

    Usage

    It’s recommended to import din prefixed whenever possible:

    import 'package:din/din.dart' as din;

    Platforms

    Currently this library is only supported on the standalone VM and Flutter, but can be easily expanded by implementing a custom HttpClient. See the VM implementation at lib/platform/vm.dart for an example. You can then pass the client in:

    void main() {
      final client = const din.ApiClient(
        rest: const din.RestClient(
          auth: const din.AuthScheme.asBot('YOUR_TOKEN_HERE'),
          http: const CustomHttpClient(),
        ),
      );
    }
    
    class CustomHttpClient implements din.HttpClient { /* TODO: Implement. */ }

    Contributing

    Discord API Changes

    As explained below, the REST API endpoints for Discord are manually specified but much/almost all of the boilerplate code around JSON encoding and generating URLs is handled by offline code generation, and published as part of this package.

    Implementation:

    • tool/codegen/resource.dart
    • tool/codegen/structure.dart

    The current implementation is fairly ad-hoc, and does not yet support all the different use-cases in the Discord API. That is also OK; the API is flexible in that some of the methods can be implemented by hand if needed, and the others are generated.

    If you make a change to anything in lib/src/schema/** or the code generators re-run the build script at tool/build.dart to update <file>.g.dart files:

    $ dart tool/build.dart

    Or, leave a watcher open to automatically rebuild:

    $ dart tool/build.dart --watch

    Testing

    The easiest way to run all the unit tests with the precise configuration needed is to run tool/test.dart. This in turn runs a series of web servers (as needed) for local testing, and pub run test to execute the test cases:

    $ dart tool/test.dart

    For manual testing, (i.e. running/debugging a specific test):

    • Run tool/servers/static.dart before running any HTTP client tests:
    $ dart tool/servers/static.dart
    
    ...
    
    $ pub run test test/clients/http_client_vm_test.dart

    End-to-end testing

    Sometimes when changing the API, or releasing a new version of this library it is important to verify that the entire end-to-end story still works connected to the real Discord API server.

    Tests in the e2e/** folder do this, but are part of tool/test.dart in order to avoid overloading Discord’s servers from continuous integration systems like travis. In order to run manually, or on something like a cron-job, run:

    $ DISCORD_API_TOKEN='1234' DISCORD_CHANNEL_ID='1234' DISCORD_USER_NAME='...' pub run test e2e

    Note that all variables above must be set as an environment variable to use these tests. If you do not have one, login to discord and create an application. Do not share this token with others, it should remain private. Make sure to add access for your bot to connect and interact with the channel specified by DISCORD_CHANNEL_ID.

    You may optionally add a config.yaml file to e2e/config.yaml instead; it is ignored by .gitignore and is for convenience while developing locally.

    api_token:  "..."
    channel_id: "..."
    user_name:  "..."

    Design

    The din package is built to be layered, customizable, and easily hackable, both for direct contributors to the package, and for packages built on top of din. There are three “tiers” of APIs available, each with high-level functionality:

    HTTP

    Din provides a minimal, platform-independent HTTP interface, or HttpClient:

    abstract class HttpClient {
      /// Sends an HTTP request to [path] using [method].
      ///
      /// May optionally define a [payload] and HTTP [headers].
      ///
      /// Unlike a standard [Future], a [CancelableOperation] may be cancelled if
      /// the in-flight request is no longer valid or wanted. In that event, the
      /// [CancelableOperation.value] may never complete.
      CancelableOperation<Map<String, Object>> requestJson(
        String path, {
        String method: 'GET',
        Map<String, Object> payload,
        Map<String, String> headers,
      });
    }

    While this itself is not that useful for building a bot or application, it does provide a very simple way to create custom HTTP implementations, such as those that do caching, offline support, or work on a variety of platforms. The built-in/default implementation works on the standalone Dart VM and Flutter.

    REST

    The official Discord API is REST-based, and requires a series of HTTP headers in order to communicate. This is encapsulated as RestClient, which in turn can make an HTTP request to any given REST endpoint. It does not have knowledge of the precise endpoints of the Discord API, though, and only communicates in raw/untyped JSON.

    Creating one is required to use ApiClient, the highest-level API provided:

    final rest = const din.RestClient(
      auth: const din.AuthScheme.asBot('YOUR_TOKEN_HERE'),
    );

    For clients or libraries that do not want to use ApiClient, they can use RestClient in order to remove much of the boiler-plate around how to connect and utilize the REST API.

    API

    A semi-automatically updated high-level API for communicating with precise REST endpoints in the Discord API used strongly-typed methods, returning strongly typed Dart objects. There is no official schema provided by the Discord team, so instead the schema is hand-maintained as metadata annotations, and the exact API is generated from that.

    See tool/build.dart, and lib/src/schema/** for more information.

    Visit original content creator repository https://github.com/matanlurey/din
  • din

    din

    An unofficial Dart library for Discord.

    Uses Dart 1.25.0-devLicense Pub Package Build Status

    GitHub Issues GitHub Forks GitHub Stars

    Install

    The current pre-release of din does not require other runtime dependencies.

    dependencies:
      din: ^0.1.0-alpha+5

    To get started, it’s recommended to read the Discord documentation and create an application for API access.

    Usage

    It’s recommended to import din prefixed whenever possible:

    import 'package:din/din.dart' as din;

    Platforms

    Currently this library is only supported on the standalone VM and Flutter, but can be easily expanded by implementing a custom HttpClient. See the VM implementation at lib/platform/vm.dart for an example. You can then pass the client in:

    void main() {
      final client = const din.ApiClient(
        rest: const din.RestClient(
          auth: const din.AuthScheme.asBot('YOUR_TOKEN_HERE'),
          http: const CustomHttpClient(),
        ),
      );
    }
    
    class CustomHttpClient implements din.HttpClient { /* TODO: Implement. */ }

    Contributing

    Discord API Changes

    As explained below, the REST API endpoints for Discord are manually specified but much/almost all of the boilerplate code around JSON encoding and generating URLs is handled by offline code generation, and published as part of this package.

    Implementation:

    • tool/codegen/resource.dart
    • tool/codegen/structure.dart

    The current implementation is fairly ad-hoc, and does not yet support all the different use-cases in the Discord API. That is also OK; the API is flexible in that some of the methods can be implemented by hand if needed, and the others are generated.

    If you make a change to anything in lib/src/schema/** or the code generators re-run the build script at tool/build.dart to update <file>.g.dart files:

    $ dart tool/build.dart

    Or, leave a watcher open to automatically rebuild:

    $ dart tool/build.dart --watch

    Testing

    The easiest way to run all the unit tests with the precise configuration needed is to run tool/test.dart. This in turn runs a series of web servers (as needed) for local testing, and pub run test to execute the test cases:

    $ dart tool/test.dart

    For manual testing, (i.e. running/debugging a specific test):

    • Run tool/servers/static.dart before running any HTTP client tests:
    $ dart tool/servers/static.dart
    
    ...
    
    $ pub run test test/clients/http_client_vm_test.dart

    End-to-end testing

    Sometimes when changing the API, or releasing a new version of this library it is important to verify that the entire end-to-end story still works connected to the real Discord API server.

    Tests in the e2e/** folder do this, but are part of tool/test.dart in order to avoid overloading Discord’s servers from continuous integration systems like travis. In order to run manually, or on something like a cron-job, run:

    $ DISCORD_API_TOKEN='1234' DISCORD_CHANNEL_ID='1234' DISCORD_USER_NAME='...' pub run test e2e

    Note that all variables above must be set as an environment variable to use these tests. If you do not have one, login to discord and create an application. Do not share this token with others, it should remain private. Make sure to add access for your bot to connect and interact with the channel specified by DISCORD_CHANNEL_ID.

    You may optionally add a config.yaml file to e2e/config.yaml instead; it is ignored by .gitignore and is for convenience while developing locally.

    api_token:  "..."
    channel_id: "..."
    user_name:  "..."

    Design

    The din package is built to be layered, customizable, and easily hackable, both for direct contributors to the package, and for packages built on top of din. There are three “tiers” of APIs available, each with high-level functionality:

    HTTP

    Din provides a minimal, platform-independent HTTP interface, or HttpClient:

    abstract class HttpClient {
      /// Sends an HTTP request to [path] using [method].
      ///
      /// May optionally define a [payload] and HTTP [headers].
      ///
      /// Unlike a standard [Future], a [CancelableOperation] may be cancelled if
      /// the in-flight request is no longer valid or wanted. In that event, the
      /// [CancelableOperation.value] may never complete.
      CancelableOperation<Map<String, Object>> requestJson(
        String path, {
        String method: 'GET',
        Map<String, Object> payload,
        Map<String, String> headers,
      });
    }

    While this itself is not that useful for building a bot or application, it does provide a very simple way to create custom HTTP implementations, such as those that do caching, offline support, or work on a variety of platforms. The built-in/default implementation works on the standalone Dart VM and Flutter.

    REST

    The official Discord API is REST-based, and requires a series of HTTP headers in order to communicate. This is encapsulated as RestClient, which in turn can make an HTTP request to any given REST endpoint. It does not have knowledge of the precise endpoints of the Discord API, though, and only communicates in raw/untyped JSON.

    Creating one is required to use ApiClient, the highest-level API provided:

    final rest = const din.RestClient(
      auth: const din.AuthScheme.asBot('YOUR_TOKEN_HERE'),
    );

    For clients or libraries that do not want to use ApiClient, they can use RestClient in order to remove much of the boiler-plate around how to connect and utilize the REST API.

    API

    A semi-automatically updated high-level API for communicating with precise REST endpoints in the Discord API used strongly-typed methods, returning strongly typed Dart objects. There is no official schema provided by the Discord team, so instead the schema is hand-maintained as metadata annotations, and the exact API is generated from that.

    See tool/build.dart, and lib/src/schema/** for more information.

    Visit original content creator repository https://github.com/matanlurey/din
  • NHANES-MetS

    Visit original content creator repository
    https://github.com/zhaoliang0302/NHANES-MetS

  • NFE1_2_3_TurtleNeck

    Turtleneck Blog Project

    프로젝트 소개

    Turtleneck Blog는 개발자들을 위한 테크 블로그 플랫폼입니다. 사용자는 마크다운 에디터를 통해 쉽게 글을 작성하고, 다른 사용자들과 좋아요 및 댓글을 통해 피드백과 소통할 수 있습니다. 이 플랫폼은 전문적인 기술 포스트를 쉽게 공유하고, 코드 및 기술에 대해 논의할 수 있는 공간을 제공합니다. 블로그는 단순한 정보 공유를 넘어서, 개발자 간의 소통과 피드백을 통해 서로의 성장을 도울 수 있는 커뮤니티를 지향합니다. 사용자는 포스트를 작성하고, 다른 사용자로부터 피드백을 받음으로써 자신의 기술적 역량을 키우고 더 나은 글을 작성하는 기회를 얻게 됩니다.

    주요 기능

    • 마크다운을 통한 간편한 글 작성: react-markdown-editor-litemarkdown-it을 사용해 쉽게 포스트를 작성할 수 있습니다. 마크다운을 이용하면 코드 블록, 표, 리스트 등 다양한 형식을 간편하게 작성할 수 있으며, 직관적인 편집 경험을 제공합니다.
    • 좋아요 및 댓글 기능: 각 게시물에 대해 좋아요와 댓글을 남길 수 있어, 사용자 간의 활발한 소통이 가능합니다. 이를 통해 작성자는 자신의 포스트에 대한 피드백을 직접 확인하고, 이를 바탕으로 글의 품질을 높일 수 있습니다.
    • 관리자 기능: 카테고리 관리, 댓글 관리, 블로그 정보 관리 기능을 통해 블로그를 효율적으로 관리할 수 있습니다. 관리자는 사용자 활동을 모니터링하고, 원활한 블로그 운영을 위해 콘텐츠를 유지하고 관리할 수 있습니다.

    기술 스택

    • 프론트엔드: React, Sass, react-markdown-editor-lite, markdown-it을 사용하여 사용자 친화적인 인터페이스와 스타일을 구현했습니다. 반응형 디자인을 적용하여 다양한 기기에서 최적의 사용자 경험을 제공합니다.
    • 백엔드: Next.js, NextAuth(JWT 사용), Mongoose, Bcrypt를 사용하여 데이터 관리와 인증 기능을 강화했습니다. 효율적인 서버 사이드 렌더링을 통해 성능을 최적화하고, JWT를 사용한 보안 강화를 도모했습니다.
    • 보안: 인증 및 보안 강화를 위해 next-auth와 Bcrypt를 사용하였습니다. 비밀번호는 안전하게 해싱되어 저장되며, 사용자 데이터는 보호됩니다.
    • 데이터베이스: MongoDB, Mongoose를 사용한 DB 연결 및 설정을 통해 확장성 높은 데이터 구조를 지원합니다. 데이터베이스는 게시글, 사용자, 댓글 등을 효율적으로 관리할 수 있도록 설계되었습니다.

    폴더 구조

    NFE1_2_3_TurtleNeck/
    ├── client/
    │   ├── public/
    │   │   ├── images/
    │   ├── src/
    │   │   ├── app/
    │   │   │   ├── admin/
    │   │   │   ├── api/
    │   │   │   └── auth/
    │   │   │       └── [...nextauth]/route.js
    │   │   ├── components/
    │   │   │   ├── Footer/
    │   │   │   ├── Navigation/
    │   │   │   ├── PostCard/
    │   │   │   └── MainPostCard/
    │   │   ├── utils/
    │   │   │   └── api.js
    └── server/
        ├── app/
        │   ├── api/
        │   │   ├── admin/
        │   │   ├── auth/
        │   │   ├── category/
        │   │   ├── comment/
        │   │   ├── like/
        │   │   ├── post/
        │   │   └── upload/
        │   └── db/
        │       ├── models/
        │       └── dbConnect.js   
        └── public/
            └── uploads/
    

    위와 같은 폴더 구조를 통해 클라이언트와 서버 코드가 명확하게 분리되어 있으며, 유지 보수와 기능 확장이 용이하도록 설계되었습니다. 각 폴더는 역할에 따라 세분화되어 있으며, 파일 경로를 직관적으로 알 수 있습니다.

    프로젝트 진행 과정

    1. 기획/설계 (10/14 – 10/23): 주제 선정, 요구사항 정의서 작성, 프로젝트 구성도, 클래스 및 시퀀스 다이어그램 작성. 이 단계에서는 사용자의 요구를 정확히 분석하고, 이를 토대로 프로젝트의 전반적인 구조를 설계했습니다. 모든 팀원이 프로젝트의 목표와 역할을 명확히 이해할 수 있도록 다양한 도구를 활용해 문서를 작성했습니다.
    2. 개발/병합 (10/23 – 11/05): 공통 컴포넌트 구현, 페이지 화면 구현, 인증, 인가, 보안 처리 및 DB 설정을 진행했습니다. 각 팀원이 맡은 역할에 따라 기능을 구현하고, 이를 병합하여 전체적인 시스템의 동작을 확인했습니다. 버그 수정 및 코드 리뷰를 통해 품질을 높였습니다.
    3. 종료/발표 (11/05 – 11/06): 최종 코드 병합 및 발표 준비. 프로젝트의 마지막 단계로, 최종적인 버그를 수정하고 안정성을 확보했습니다. 발표 자료를 준비하여 프로젝트의 결과물과 과정을 공유했습니다.

    팀 구성

    • 프론트엔드: 이동인, 김혜준
      • 상세페이지, 카드 컴포넌트, 메인페이지, 관리자 CSS 구현
      • 사용자의 요구에 맞는 인터페이스를 구축하고, CSS를 통해 사용자 경험을 개선하였습니다.
    • 백엔드: 정해량, 최수진
      • 보안 설정, 클라이언트 API, 이미지 처리, 데이터베이스 설정
      • 데이터의 안전한 저장과 효율적인 관리를 위해 다양한 백엔드 기술을 활용하였습니다.

    페이지 구성

    • 메인 페이지: 네비게이션, 블로그 이름 & 설명, 최신 게시글, 게시글 카드 목록, 페이지네이션, 푸터로 구성되어 있습니다. 메인 페이지는 블로그의 첫 인상을 결정짓는 중요한 요소로, 사용자 친화적인 디자인과 최신 정보 제공을 목표로 했습니다.
    • 상세 페이지: 게시글 제목, 메타 데이터, 게시글 상세 내용, 댓글 기능을 포함하여 사용자가 포스트를 깊이 있게 이해하고 소통할 수 있도록 설계되었습니다. 댓글 기능을 통해 사용자는 직접적인 의견을 남길 수 있습니다.
    • 게시글 작성/수정 페이지: 카테고리, 제목, 태그, 썸네일 입력 및 마크다운 에디터 기능을 제공하여, 사용자가 손쉽게 게시글을 작성하고 수정할 수 있도록 돕습니다. 미리보기 기능을 통해 작성 중인 글의 형태를 즉시 확인할 수 있습니다.
    • 관리 페이지: 블로그 정보 관리, 카테고리 관리, 댓글 관리 기능이 있으며, 관리자는 이를 통해 블로그의 콘텐츠와 사용자 활동을 체계적으로 관리할 수 있습니다.

    설치 및 실행 방법

    1. 저장소를 클론합니다.
      git clone [저장소 URL]
    2. 클라이언트 및 서버 디렉토리로 이동하여 필요한 패키지를 설치합니다.

      cd client && npm install
      cd ../server && npm install
    3. 환경 변수 파일을 설정합니다 (.env.local 파일).
    4. 개발 서버를 실행합니다.
      npm run dev
    5. 서버와 클라이언트가 성공적으로 연결되면, 로컬 환경에서 프로젝트를 테스트할 수 있습니다. 각 기능이 의도한 대로 작동하는지 확인하고, 문제가 발생할 경우 디버깅을 통해 해결합니다.

    라이선스

    MIT License를 따릅니다. 누구나 자유롭게 이 프로젝트를 사용하고 수정할 수 있으며, 이에 대한 제약 사항은 MIT 라이선스에 명시되어 있습니다.

    기여 방법

    기여를 원하시는 분은 이슈를 생성하거나 풀 리퀘스트를 제출해 주세요. 항상 환영합니다! 프로젝트에 기여함으로써 더 나은 플랫폼을 함께 만들어 나갈 수 있습니다. 버그 신고, 기능 제안, 코드 개선 등 다양한 형태의 기여가 가능합니다.

    Visit original content creator repository
    https://github.com/prgrms-fe-devcourse/NFE1_2_3_TurtleNeck

  • Python_Tutorial_For_Data-Science

    Python Tutorial for Data Science

    This repository is a comprehensive guide for learning data science using Python. It covers various essential libraries and tools commonly used in the field of data science, including Jupyter Notebook, Matplotlib, NumPy, Pandas, Scikit-learn, and PyTorch.

    Contents

    1. Jupyter Notebook: Introduction to Jupyter Notebook and its features.

    2. Matplotlib: Exploring data visualization using Matplotlib library.

    3. NumPy: Introduction to NumPy and its powerful array operations for numerical computing.

    4. Pandas: Understanding data manipulation and analysis with Pandas.

    5. Scikit-learn: Introduction to machine learning with Scikit-learn library.

    6. PyTorch: Getting started with deep learning using PyTorch framework.

    Usage

    1. Clone the repository using the following command:

      bashCopy code

      git clone https://github.com/NagiPragalathan/Python_Tutorial_For_Data-Science.git

    2. Install the required dependencies using pip:

      Copy code

      pip install -r requirements.txt

    3. Navigate to the desired notebook in the notebooks directory and open it using Jupyter Notebook.

    4. Follow the instructions and code examples provided in each notebook to learn and practice data science concepts.

    Contributing

    If you find any issues or have suggestions for improvement, please feel free to contribute to this repository. You can submit bug reports, feature requests, or pull requests to help enhance the learning experience for everyone.

    License

    This project is licensed under the MIT License. Feel free to use and modify the code for educational purposes.

    Let’s dive into the exciting world of data science and explore the power of Python together!

    Visit original content creator repository
    https://github.com/NagiPragalathan/Python_Tutorial_For_Data-Science

  • cc1101-tool

    cc1101-tool

    RF tool based on CC1101 module and Arduino Pro Micro 8VMHz/3.3V. Allows using CLI to control CC1101 board over USB interface. Putty or any other serial terminal can be used. It has similar functionality to YardStick One but is cheaper and does not need specialized software. Allows for RF jamming and replay attacks as well. It has RAW recording/replaying function which works exactly the same as in the Flipper Zero. Additional function is Radio Chat communicator

    You simply connect your Arduino Pro Micro (Arduino Leonardo clone from Sparkfun) to USB port of your PC and launch Putty terminal to communicate with CC1101 module over USB Serial port ( /dev/ttyACM0 port in Linux, COMxx in Windows).

    Also you may connect this device to Android OTG USB port in your smartphone for portable hacking and use USB Serial Terminal application with option CDC driver set to communicate with the device ( app : https://play.google.com/store/apps/details?id=de.kai_morich.serial_usb_terminal ). When using Serial Terminal app on Android first go to the Settings in the app on your smartphone then upper side of the screen select “Send” and then set “Newline” as CR only. It is sending to many characters to the device and extra character of Newline sometimes stops some of commands like “rxraw” for example.

    Following commands are available :

    setmodulation <mode>         // set modulation mode. 0 = 2-FSK, 1 = GFSK, 2 = ASK/OOK, 3 = 4-FSK, 4 = MSK. 
    
    setmhz <frequency>           // Here you can set your basic frequency. default = 433.92).The cc1101 can: 300-348 MHZ, 387-464MHZ and 779-928MHZ.
    
    setdeviation <deviation>     // Set the Frequency deviation in kHz. Value from 1.58 to 380.85. Default is 47.60 kHz.
    
    setchannel <channel>         // Set the Channelnumber from 0 to 255. Default is cahnnel 0.
    
    setchsp <spacing>            // The channel spacing is multiplied by the channel number CHAN and added to the base frequency in kHz. Value from 25.39 to 405.45. Default is 199.95 kHz. 
    
    setrxbw <Receive bandwidh>   // Set the Receive Bandwidth in kHz. Value from 58.03 to 812.50. Default is 812.50 kHz.
    
    setdrate <datarate>          // Set the Data Rate in kBaud. Value from 0.02 to 1621.83. Default is 99.97 kBaud!
    
    setpa <power value>          // Set TxPower. The following settings are possible depending on the frequency band.  (-30  -20  -15  -10  -6    0    5    7    10   11   12) Default is max!
    
    setsyncmode  <sync mode>     // Combined sync-word qualifier mode. 0 = No preamble/sync. 1 = 16 sync word bits detected. 2 = 16/16 sync word bits detected. 3 = 30/32 sync word bits detected. 4 = No preamble/sync, carrier-sense above threshold. 5 = 15/16 + carrier-sense above threshold. 6 = 16/16 + carrier-sense above threshold. 7 = 30/32 + carrier-sense above threshold.
    
    setsyncword <LOW, HIGH>      // Set sync word. Must be the same for the transmitter and receiver. (Syncword high, Syncword low)
    
    setadrchk <address check>    // Controls address check configuration of received packages. 0 = No address check. 1 = Address check, no broadcast. 2 = Address check and 0 (0x00) broadcast. 3 = Address check and 0 (0x00) and 255 (0xFF) broadcast.
    
    setaddr <address>            // Address used for packet filtration. Optional broadcast addresses are 0 (0x00) and 255 (0xFF).
    
    setwhitedata <whitening>     // Turn data whitening on / off. 0 = Whitening off. 1 = Whitening on.
    
    setpktformat <pkt format>    // Format of RX and TX data. 0 = Normal mode, use FIFOs for RX and TX. 1 = Synchronous serial mode, Data in on GDO0 and data out on either of the GDOx pins. 2 = Random TX mode; sends random data using PN9 generator. Used for test. Works as normal mode, setting 0 (00), in RX. 3 = Asynchronous serial mode, Data in on GDO0 and data out on either of the GDOx pins.
    
    setlengthconfig <mode>       // 0 = Fixed packet length mode. 1 = Variable packet length mode. 2 = Infinite packet length mode. 3 = Reserved 
    
    setpacketlength <mode>       // Indicates the packet length when fixed packet length mode is enabled. If variable packet length mode is used, this value indicates the maximum packet length allowed.
    
    setcrc <mode>                // 1 = CRC calculation in TX and CRC check in RX enabled. 0 = CRC disabled for TX and RX.
    
    setcrcaf <mode>             // Enable automatic flush of RX FIFO when CRC is not OK. This requires that only one packet is in the RXIFIFO and that packet length is limited to the RX FIFO size.
    
    setdcfilteroff <mode>        // Disable digital DC blocking filter before demodulator. Only for data rates ≤ 250 kBaud The recommended IF frequency changes when the DC blocking is disabled. 1 = Disable (current optimized). 0 = Enable (better sensitivity).
    
    setmanchester <mode>         // Enables Manchester encoding/decoding. 0 = Disable. 1 = Enable.
    
    setfec <mode>                // Enable Forward Error Correction (FEC) with interleaving for packet payload (Only supported for fixed packet length mode. 0 = Disable. 1 = Enable.
    
    setpre <mode>                // Sets the minimum number of preamble bytes to be transmitted. Values: 0 : 2, 1 : 3, 2 : 4, 3 : 6, 4 : 8, 5 : 12, 6 : 16, 7 : 24
    
    setpqt <mode>                // Preamble quality estimator threshold. The preamble quality estimator increases an internal counter by one each time a bit is received that is different from the previous bit, and decreases the counter by 8 each time a bit is received that is the same as the last bit. A threshold of 4∙PQT for this counter is used to gate sync word detection. When PQT=0 a sync word is always accepted.
    
    setappendstatus <mode>       // When enabled, two status bytes will be appended to the payload of the packet. The status bytes contain RSSI and LQI values, as well as CRC OK.
    
    getrssi                      // Shows radio quality information about last received RF data frame.
    
    scan <start freq> <end freq> // Scan frequency range for the highest signal and display results
    
    rx                           // Enable or disable printing of received RF packets on serial terminal.
    
    tx  <hex-vals>               // Send the packet of max 60 bytes < hex values > hex values over RF 
    
    jam                          // Enable or disable continous jamming on selected band with selected modulation etc... 
    
    brute <usec> <nb-of-bits>    // Brute force garage gate with <number-of-bits> keyword where symbol length is <microseconds>
    
    rec                          // Enable or disable recording frames in the buffer.
    
    show                         // Show content of recording buffer
    
    add <hex-vals>               // Manually add single frame payload (max 60 hex values) to the buffer so it can be replayed
    
    flush                        // Clear the recording buffer
    
    play <N>                     // Replay 0 = all frames or N-th recorded frame
    
    rxraw <microseconds>         // Sniffs radio by sampling with <microsecond> interval and prints received bytes in hex
    
    addraw <hex-vals>            // Manually add chunks (max 60 hex values) to the buffer so they can be further replayed.
    
    recraw <microseconds>        // Recording RAW RF data with <microsecond> sampling interval
    
    showraw                      // Showing content of recording buffer in RAW format
    
    showbit                      // Showing content of recording buffer in RAW format as a stream of bits.
    
    playraw <microseconds>       // Replaying previously recorded RAW RF data with <microsecond> sampling interval
    
    save                         // Store recording buffer content in non-volatile memory
    
    load                         // Load the content from non-volatile memory to the recording buffer
    
    echo <mode>                  // Enable or disable Echo on serial terminal. 1 = enabled, 0 = disabled
    
    chat                         // switching device into chat mode 
    
    x                            // Stops activities like jamming/receiving/recording packets
    
    init                         // Restarts CC1101 board with default parameters 
    

    The code uses SmartRC library (modified Electrohouse library by Little_S@tan) which allows to customize ALL transmission parameters in human readable format without using SmartRF studio from TI (CC1101 parameter customization tool). To use it please download following ZIP library from following github link https://github.com/LSatan/SmartRC-CC1101-Driver-Lib and attach it to the script in Arduino IDE.

    Arduino Pro Micro board ( ATMEGA32U4 chip ) must support 3.3Volt VCC and 3.3V TTL logic because this is required by CC1101 board, otherwise you will fry CC1101 chip. Please follow this guide to setup your Arduino environment for Arduino Pro Micro board : https://learn.sparkfun.com/tutorials/pro-micro–fio-v3-hookup-guide/all

    If you are having issues with uploading the code from Arduino IDE to the board, after pressing “Upload” in Arduino you have to immediatelly short GND+RST pins two times in few seconds. Then bootloader in Arduino Pro Micro will start (common issue) and upload will begin.

    Connections to be made for ARDUINO PRO MICRO :

    ARDUINO PRO MICRO 3.3V / 8MHz <-> CC1101 BOARD

    DIGITAL PIN 3 ( PD0 / INT0 ) <-> CC1101 GDO0

    DIGITAL PIN 9 ( PB5 ) <-> CC1101 GDO2

    DIGITAL PIN 10 ( PB6 ) <-> CC1101 CSN / CS / SS

    DIGITAL PIN 16 ( PB2 / MOSI ) <-> CC1101 MOSI / SI

    DIGITAL PIN 14 ( PB3 / MISO ) <-> CC1101 MISO / SO

    DIGITAL PIN 15 ( PB1 / SCK ) <-> CC1101 SCLK / CLK

    VCC 3.3V <-> CC1101 VCC

    GND <-> CC1101 GND


    If you want to use different Arduino Board, please change pin assignment in the beginning of the source code and adjust size of EEPROM/FLASH for storing recorded data and size of SRAM memory


    Example for ESP32 board :

    #define RECORDINGBUFFERSIZE 4096 // Buffer for recording the frames

    #define EPROMSIZE 512 // Size of EEPROM in your Arduino chip. For ESP32 it is Flash simulated 512 bytes only

    // defining PINs set for ESP32 module

    byte sck = 18; // GPIO 18

    byte miso = 19; // GPIO 19

    byte mosi = 23; // GPIO

    byte ss = 5; // GPIO 5

    int gdo0 = 2; // GPIO 2

    int gdo2 = 4; // GPIO 4


    Example for XIAO ESP32 C3 – ATTENTION ! This board may require some shielding of cables connected to GDO0 pin when using some cheapest CC1101 boards (green D-SUN f.ex.) to properly work with RXRAW/RECRAW commands. It catches noise like “FF” in RXRAW/RECRAW.

    #define RECORDINGBUFFERSIZE 4096 // Buffer for recording the frames

    #define EPROMSIZE 512 // Size of EEPROM in your Arduino chip. For ESP32 it is Flash simulated 512 bytes only

    // defining PINs set for XIAO ESP32 C3

    byte sck = 8; // GPIO 8

    byte miso = 4; // GPIO 4

    byte mosi = 10; // GPIO 10

    byte ss = 20; // GPIO 20

    int gdo0 = 21; // GPIO 21

    int gdo2 = 7; // GPIO 7


    Example for WEMOS D1 MINI module

    #define RECORDINGBUFFERSIZE 4096 // Buffer for recording the frames

    #define EPROMSIZE 4096 // Size of EEPROM in your Arduino chip. For ESP8266 size is 4096

    // defining PINs set for ESP8266 – WEMOS D1 MINI module

    byte sck = 14; // GPIO 14

    byte miso = 12; // GPIO 12

    byte mosi = 13; // GPIO 13

    byte ss = 15; // GPIO 15

    int gdo0 = 5; // GPIO 5

    int gdo2 = 4; // GPIO 4


    Example for Arduino Nano board – ATTENTION ! I HAVE TESTED THIS BOARD AND IT REQUIRES TTL LOGIC COVERTER 5V<->3.3V TXS0108E ESPECIALLY FOR BOARD CC1101 : E07-M1101D, otherwise it does not work

    #define RECORDINGBUFFERSIZE 1024 // Buffer for recording the frames

    #define EPROMSIZE 1024 // Size of EEPROM in your Arduino chip.

    // defining PINs for Arduino NANO

    byte sck = 13; // D13

    byte miso = 12; // D12

    byte mosi = 11; // D11

    byte ss = 10; // D10

    byte gdo0 = 6; // D6

    byte gdo2 = 2; // D2


    Example for Arduino Pro MINI board – ATTENTION ! I HAVE TESTED THIS BOARD AND IT REQUIRES TTL LOGIC COVERTER 5V<->3.3V TXS0108E ESPECIALLY FOR BOARD CC1101 : E07-M1101D, otherwise it does not work

    #define RECORDINGBUFFERSIZE 1024 // Buffer for recording the frames

    #define EPROMSIZE 1024 // Size of EEPROM in your Arduino chip.

    // defining PINs for Arduino PRO MINI

    byte sck = 16; // D13

    byte miso = 15; // D12

    byte mosi = 14; // D11

    byte ss = 13; // D10

    byte gdo0 = 9; // D6

    byte gdo2 = 5; // D2


    Example for Raspberry Pi Pico / RP2040 board – ATTENTION ! I HAVE TESTED THIS BOARD AND IT USES 3.3V LOGIC

    #define RECORDINGBUFFERSIZE 4096 // Buffer for recording the frames

    #define EPROMSIZE 512 // Size of EEPROM in your Arduino chip.

    // defining PINs for Raspberry Pi Pico

    // see pinout: https://cdn-learn.adafruit.com/assets/assets/000/099/339/original/raspberry_pi_Pico-R3-Pinout-narrow.png

    byte sck = 2;

    byte miso = 4;

    byte mosi = 3;

    byte ss = 5;

    int gdo0 = 7;

    int gdo2 = 6;

    Attention ! There may be different SPI pins set for your Pico board :

    byte miso = 16;

    byte ss = 17;

    byte mosi = 19;

    byte sck = 18;


    First version of this project was presented in this video : https://youtu.be/iPVckkTjsd0
    Using Universal Radio Hacker and my CC1101-tool is presented in following video : https://youtu.be/mdkEK_wmWJA

    Change log :

    08.06.2023 : optimized CLI

    • removed unnecessary parameters for commands RX, TX, JAM.
    • changed command JAMM to JAM.
    • optimized output of RX command – now will print directly hex values with no description when sniffer enabled.
    • corrected reaction for CR/LF when using with “Serial Terminal” application on USB OTG port on Android phones
    • Added CHAT mode, if you have couple of these devices you may use it as and IRC like communicator on selected band/modulation/frequency/channel…

    09.06.2023 : added RAW mode as in Flipper Zero

    • rxraw “interval microseconds”,
    • recraw “interval usec”,
    • playraw “interval usec”,
    • showraw – for record & replay attacking.
    • buffer of 1536 bytes is used to store recording (in ATMEGA32U4, 1024 for Atmega Mega/Uno/Nano, 4096 or more for ESP32 boards).
    • after playing with RAW mode please always enter “init” command to restart CC1101 chip. Don’t worry about Low Memory warning during Arduino compilation it will work JUST FINE.. Enjoy 🙂

    10.06.2023 :

    • added Arduino Mega/Nano/Uno version which requires TTL logic converter for 3.3V – TXS0108E.
    • added ESP32 version.
    • changed RECRAW command to start recording RAW signal once something appears over the radio.
    • added command ADDRAW to enable manual composition of the signal in the buffer (by copying hex number chunks from Universal Radio Hacker tool for example).
    • added option SCAN to find a peak frequency for recording/jamming..

    17.06.2023 :

    • added SAVE function to store recorded frames buffer into non-volatile EEPROM memory of the Arduino chip
    • added LOAD function to restore recorded frames from non-volatile memory and put them into recording buffer for replaying.
    • added SHOWBIT command to display RAW data from the buffer as stream of bits.
    • corrected ESP32 version which has problem with changing (char *) type to (byte *) due to different C++ compiler for ESP32 boards

    18.06.2023 :

    • updated bit storage order in PLAYRAW, RXRAW, RECRAW commands to match the type used in Universal Radio Hacker tool : https://github.com/jopohl/urh

    30.06.2023

    • added debug message during CC1101 startup
    • corrected EEPROM usage for ESP32 chip based Arduino – ESP32 has 512 bytes, ESP8266 has 4096 bytes
    • added ESP32-WROOM version ( I have tested it succesfully with my own board )
    • added ESP8266 – WEMOS D1 Mini version

    08.07.2023

    • corrected ESP8266 version, WDT watchdog restarts MOSTLY solved (this single core chip is heavy loaded with internal WiFI procedures and TCP IP stack). Code was successfuly tested on WEMOS D1 MINI clone and D-SUN CC1101 board. The advantage of using WEMOS D1 MINI biggest size of FLASH simulated EEPROM for RF sequences storage. ESP32 chips are dual core and my code runs better.

    13.07.2023

    • changed default data rate for Packet Mode from 1.2Kbaud to 9.6Kbaud which removes problems with Watchdog Restart on ESP8266 board and improves stability

    27.07.2023:

    • rp2040 board added

    18.08.2023:

    • added command BRUTE for brute force attack on some DIP switches based garage gates. Sometimes the code hangs after executing full brute force cycle. Trying to find the root cause… Another bad news is that I have reached full FLASH capacity of ATMEGA32U4 so no more extensions are possible to the code for this chip.

    02.09.2023

    • WIFI client mode for ESP8266 board added – there is a separate source code version wifi in the name. Before uploading the code you need to assign an IP address to the module , put correct default gateway as well as configure SSID of your WIFI router and the WIFI password in the code like below (defaults are for Android tethering access point). Wifi client mode can be used to extend widely the range between CC1101 device and you PC/smartphone which is used to control this board. Beetween them you have external Access Point that will be “man in the middle” to extend the WIFI range…

    • The ESP8266 board connects to your WIFI router/access point (you need 2nd mobile phone which will serve as an accesspoint for your own phone and ESP8266 board) and you do a TELNET to its IP address 192.168.43.100 from the other smartphone :

      IPAddress ip(192, 168, 43, 100); // Local Static IP address

      IPAddress gateway(192, 168, 43, 1); // Gateway IP address

      IPAddress subnet(255, 255, 255, 0); // Subnet Mask

      const char ssid[] = “AndroidAP”; // Change to your Router SSID

      const char password[] = “password”; // Change to your Router Password

    Example scenario for WIFI / telnet connection when ESP8266 is WIFI client :
    ESP8266 + CC1101 WIFI client at 192.168.43.100 <-> Smartphone #1 wifi tethering / wifi access point at 192.168.43.1 <-> Smartphone #2 WIFI client / Connectbot – telnet to 192.168.43.100.
    ATTENTION ! When using WIFI versions I recommend to set ESP8266 CPU speed to 160 MHz : in arduino IDE – go to Tools, in dropdown list select “CPU Frequency 160MHz” instead of “CPU Frequency 80MHz”.

    08.09.2023

    • WIFI Access Point mode to ESP8266 board added – there is a separate source code version with wifi-ap in the name. Before uploading the code you can change an IP address to the module and SSID for your ESP8266 board. Default is SSID “cc1101” and IP address for telnet “192.168.1.100”. This is the simplest scenario, use Connectbot and Telnet protocol to connect to CC1101 board over TCP port 23. ATTENTION ! When using WIFI versions I recommend to set ESP8266 CPU speed to 160 MHz : in arduino IDE – go to Tools, in dropdown list select “CPU Frequency 160MHz” instead of “CPU Frequency 80MHz”.

    22.11.2023
    Corrected bug in showbit() function, all the thanks go to jps1x2.

    08.02.2024
    Corrected bug in scan() function (not accepting MHz fractions in frequency range), all the thanks go to chris4soft

    Known Bugs : sometimes RX command does not work correctly after many big frames have been received (in packet mode, not in async mode). This may be due to some memory leak in SmartRC library. Still checking what is the reason. Keep attention to putting an argument to rxraw, playraw command – otherwise ESP8266 are restarting themselves when wifi in use (stack overflow).

    Visit original content creator repository
    https://github.com/mcore1976/cc1101-tool

  • triple-buffer

    Triple buffering in Rust

    MPLv2 licensed On crates.io On docs.rs Continuous Integration Requires rustc 1.74.0+

    What is this?

    This is an implementation of triple buffering written in Rust. You may find it useful for the following class of thread synchronization problems:

    • There is one producer thread and one consumer thread
    • The producer wants to update a shared memory value periodically
    • The consumer wants to access the latest update from the producer at any time

    For many use cases, you can use the ergonomic write/read interface, where the producer moves values into the buffer and the consumer accesses the latest buffer by shared reference:

    // Create a triple buffer
    use triple_buffer::triple_buffer;
    let (mut buf_input, mut buf_output) = triple_buffer(&0);
    
    // The producer thread can move a value into the buffer at any time
    let producer = std::thread::spawn(move || buf_input.write(42));
    
    // The consumer thread can read the latest value at any time
    let consumer = std::thread::spawn(move || {
        let latest = buf_output.read();
        assert!(*latest == 42 || *latest == 0);
    });
    
    // Wait for both threads to be done
    producer.join().unwrap();
    consumer.join().unwrap();

    In situations where moving the original value away and being unable to modify it on the consumer’s side is too costly, such as if creating a new value involves dynamic memory allocation, you can use a lower-level API which allows you to access the producer and consumer’s buffers in place and to precisely control when updates are propagated:

    // Create and split a triple buffer
    use triple_buffer::triple_buffer;
    let (mut buf_input, mut buf_output) = triple_buffer(&String::with_capacity(42));
    
    // --- PRODUCER SIDE ---
    
    // Mutate the input buffer in place
    {
        // Acquire a reference to the input buffer
        let input = buf_input.input_buffer_mut();
    
        // In general, you don't know what's inside of the buffer, so you should
        // always reset the value before use (this is a type-specific process).
        input.clear();
    
        // Perform an in-place update
        input.push_str("Hello, ");
    }
    
    // Publish the above input buffer update
    buf_input.publish();
    
    // --- CONSUMER SIDE ---
    
    // Manually fetch the buffer update from the consumer interface
    buf_output.update();
    
    // Acquire a read-only reference to the output buffer
    let output = buf_output.output_buffer();
    assert_eq!(*output, "Hello, ");
    
    // Or acquire a mutable reference if necessary
    let output_mut = buf_output.output_buffer_mut();
    
    // Post-process the output value before use
    output_mut.push_str("world!");

    Finally, as a middle ground before the maximal ergonomics of the write() API and the maximal control of the input_buffer_mut()/publish() API, you can also use the input_buffer_publisher() RAII API on the producer side, which ensures that publish() is automatically called when the resulting input buffer handle goes out of scope:

    // Create and split a triple buffer
    use triple_buffer::triple_buffer;
    let (mut buf_input, _) = triple_buffer(&String::with_capacity(42));
    
    // Mutate the input buffer in place and publish it
    {
        // Acquire a reference to the input buffer
        let mut input = buf_input.input_buffer_publisher();
    
        // In general, you don't know what's inside of the buffer, so you should
        // always reset the value before use (this is a type-specific process).
        input.clear();
    
        // Perform an in-place update
        input.push_str("Hello world!");
    
        // Input buffer is automatically published at the end of the scope of
        // the "input" RAII guard
    }
    
    // From this point on, the consumer can see the updated version

    Give me details! How does it compare to alternatives?

    Compared to a mutex:

    • Only works in single-producer, single-consumer scenarios
    • Is nonblocking, and more precisely bounded wait-free. Concurrent accesses will be slowed down by cache contention, but no deadlock, livelock, or thread scheduling induced slowdown is possible.
    • Allows the producer and consumer to work simultaneously
    • Uses a lot more memory (3x payload + 3x bytes vs 1x payload + 1 bool)
    • Does not allow in-place updates, as the producer and consumer do not access the same memory location
    • Should have faster reads and slower updates, especially if in-place updates are more efficient than writing a fresh copy of the data.
      • When the data hasn’t been updated, the readout transaction of triple buffering only requires a memory read, no atomic operation, and it can be performed in parallel with any ongoing update.
      • When the data has been updated, the readout transaction requires an infaillible atomic operation, which may or may not be faster than the faillible atomic operations used by most mutex implementations.
      • Unless your data cannot be updated in place and must always be fully rewritten, the ability provided by mutexes to update data in place should make updates a lot more efficient, dwarfing any performance difference originating from the synchronization protocol.

    Compared to the read-copy-update (RCU) primitive from the Linux kernel:

    • Only works in single-producer, single-consumer scenarios
    • Has higher dirty read overhead on relaxed-memory architectures (ARM, POWER…)
    • Does not require accounting for reader “grace periods”: once the reader has gotten access to the latest value, the synchronization transaction is over
    • Does not use the compare-and-swap hardware primitive on update, which is inefficient by design as it forces its users to retry transactions in a loop.
    • Does not suffer from the ABA problem, allowing much simpler code
    • Allocates memory on initialization only, rather than on every update
    • May use more memory (3x payload + 3x bytes vs 1x pointer + amount of payloads and refcounts that depends on the readout and update pattern)
    • Should be slower if updates are rare, faster if updates are frequent
      • The RCU’s happy reader path is slightly faster (no flag to check), but its update procedure is a lot more involved and costly.

    Compared to sending the updates on a message queue:

    • Only works in single-producer, single-consumer scenarios (queues can work in other scenarios, although the implementations are much less efficient)
    • Consumer only has access to the latest state, not the previous ones
    • Consumer does not need to get through every previous state
    • Is nonblocking AND uses bounded amounts of memory (with queues, it’s a choice, unless you use one of those evil queues that silently drop data when full)
    • Can transmit information in a single move, rather than two
    • Should be faster for any compatible use case.
      • Queues force you to move data twice, once in, once out, which will incur a significant cost for any nontrivial data. If the inner data requires allocation, they force you to allocate for every transaction. By design, they force you to store and go through every update, which is not useful when you’re only interested in the latest version of the data.

    In short, triple buffering is what you’re after in scenarios where a shared memory location is updated frequently by a single writer, read by a single reader who only wants the latest version, and you can spare some RAM.

    • If you need multiple producers, look somewhere else
    • If you need multiple consumers, you may be interested in my related “SPMC buffer” work, which basically extends triple buffering to multiple consumers
    • If you can’t tolerate the RAM overhead or want to update the data in place, try a Mutex instead (or possibly an RWLock)
    • If the shared value is updated very rarely (e.g. every second), try an RCU
    • If the consumer must get every update, try a message queue

    How do I know your unsafe lock-free code is working?

    By running the tests, of course! Which is unfortunately currently harder than I’d like it to be.

    First of all, we have sequential tests, which are very thorough but obviously do not check the lock-free/synchronization part. You run them as follows:

    $ cargo test
    

    Then we have concurrent tests where, for example, a reader thread continuously observes the values from a rate-limited writer thread, and makes sure that he can see every single update without any incorrect value slipping in the middle.

    These tests are more important, but also harder to run because one must first check some assumptions:

    • The testing host must have at least 2 physical CPU cores to test all possible race conditions
    • No other code should be eating CPU in the background. Including other tests.
    • As the proper writing rate is system-dependent, what is configured in this test may not be appropriate for your machine.
    • You must test in release mode, as compiler optimizations tend to create more opportunities for race conditions.

    Taking this and the relatively long run time (~10-20 s) into account, the concurrent tests are ignored by default. To run them, make sure nothing is eating CPU in the background and do:

    $ cargo test --release -- --ignored --nocapture --test-threads=1
    

    Finally, we have benchmarks, which allow you to test how well the code is performing on your machine. We are now using criterion for said benchmarks, which seems that to run them, you can simply do:

    $ cargo install cargo-criterion
    $ cargo criterion
    

    These benchmarks exercise the worst-case scenario of u8 payloads, where synchronization overhead dominates as the cost of reading and writing the actual data is only 1 cycle. In real-world use cases, you will spend more time updating buffers and less time synchronizing them.

    However, due to the artificial nature of microbenchmarking, the benchmarks must exercise two scenarios which are respectively overly optimistic and overly pessimistic:

    1. In uncontended mode, the buffer input and output reside on the same CPU core, which underestimates the overhead of transferring modified cache lines from the L1 cache of the source CPU to that of the destination CPU.
      • This is not as bad as it sounds, because you will pay this overhead no matter what kind of thread synchronization primitive you use, so we’re not hiding triple-buffer specific overhead here. All you need to do is to ensure that when comparing against another synchronization primitive, that primitive is benchmarked in a similar way.
    2. In contended mode, the benchmarked half of the triple buffer is operating under maximal load from the other half, which is much more busy than what is actually going to be observed in real-world workloads.
      • In this configuration, what you’re essentially measuring is the performance of your CPU’s cache line locking protocol and inter-CPU core data transfers under the shared data access pattern of triple-buffer.

    Therefore, consider these benchmarks’ timings as orders of magnitude of the best and the worst that you can expect from triple-buffer, where actual performance will be somewhere inbetween these two numbers depending on your workload.

    On an Intel Core i3-3220 CPU @ 3.30GHz, typical results are as follows:

    • Clean read: 0.9 ns
    • Write: 6.9 ns
    • Write + dirty read: 19.6 ns
    • Dirty read (estimated): 12.7 ns
    • Contended write: 60.8 ns
    • Contended read: 59.2 ns

    License

    This crate is distributed under the terms of the MPLv2 license. See the LICENSE file for details.

    More relaxed licensing (Apache, MIT, BSD…) may also be negociated, in exchange of a financial contribution. Contact me for details at knights_of_ni AT gmx DOTCOM.

    Visit original content creator repository https://github.com/HadrienG2/triple-buffer