forgejo/services/repository/push.go
Jason Song 4011821c94
Implement actions (#21937)
Close #13539.

Co-authored by: @lunny @appleboy @fuxiaohei and others.

Related projects:
- https://gitea.com/gitea/actions-proto-def
- https://gitea.com/gitea/actions-proto-go
- https://gitea.com/gitea/act
- https://gitea.com/gitea/act_runner

### Summary

The target of this PR is to bring a basic implementation of "Actions",
an internal CI/CD system of Gitea. That means even though it has been
merged, the state of the feature is **EXPERIMENTAL**, and please note
that:

- It is disabled by default;
- It shouldn't be used in a production environment currently;
- It shouldn't be used in a public Gitea instance currently;
- Breaking changes may be made before it's stable.

**Please comment on #13539 if you have any different product design
ideas**, all decisions reached there will be adopted here. But in this
PR, we don't talk about **naming, feature-creep or alternatives**.

### ⚠️ Breaking

`gitea-actions` will become a reserved user name. If a user with the
name already exists in the database, it is recommended to rename it.

### Some important reviews

- What is `DEFAULT_ACTIONS_URL` in `app.ini` for?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1055954954
- Why the api for runners is not under the normal `/api/v1` prefix?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061173592
- Why DBFS?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061301178
- Why ignore events triggered by `gitea-actions` bot?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1063254103
- Why there's no permission control for actions?
  - https://github.com/go-gitea/gitea/pull/21937#discussion_r1090229868

### What it looks like

<details>

#### Manage runners

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205870657-c72f590e-2e08-4cd4-be7f-2e0abb299bbf.png">

#### List runs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872794-50fde990-2b45-48c1-a178-908e4ec5b627.png">


#### View logs

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205872501-9b7b9000-9542-4991-8f55-18ccdada77c3.png">



</details>

### How to try it

<details>

#### 1. Start Gitea

Clone this branch and [install from
source](https://docs.gitea.io/en-us/install-from-source).

Add additional configurations in `app.ini` to enable Actions:

```ini
[actions]
ENABLED = true
```

Start it.

If all is well, you'll see the management page of runners:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205877365-8e30a780-9b10-4154-b3e8-ee6c3cb35a59.png">


#### 2. Start runner

Clone the [act_runner](https://gitea.com/gitea/act_runner), and follow
the
[README](https://gitea.com/gitea/act_runner/src/branch/main/README.md)
to start it.

If all is well, you'll see a new runner has been added:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205878000-216f5937-e696-470d-b66c-8473987d91c3.png">

#### 3. Enable actions for a repo

Create a new repo or open an existing one, check the `Actions` checkbox
in settings and submit.

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879705-53e09208-73c0-4b3e-a123-2dcf9aba4b9c.png">
<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205879383-23f3d08f-1a85-41dd-a8b3-54e2ee6453e8.png">

If all is well, you'll see a new tab "Actions":

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205881648-a8072d8c-5803-4d76-b8a8-9b2fb49516c1.png">

#### 4. Upload workflow files

Upload some workflow files to `.gitea/workflows/xxx.yaml`, you can
follow the [quickstart](https://docs.github.com/en/actions/quickstart)
of GitHub Actions. Yes, Gitea Actions is compatible with GitHub Actions
in most cases, you can use the same demo:

```yaml
name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
          ls ${{ github.workspace }}
      - run: echo "🍏 This job's status is ${{ job.status }}."
```

If all is well, you'll see a new run in `Actions` tab:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884473-79a874bc-171b-4aaf-acd5-0241a45c3b53.png">

#### 5. Check the logs of jobs

Click a run and you'll see the logs:

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/9418365/205884800-994b0374-67f7-48ff-be9a-4c53f3141547.png">

#### 6. Go on

You can try more examples in [the
documents](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
of GitHub Actions, then you might find a lot of bugs.

Come on, PRs are welcome.

</details>

See also: [Feature Preview: Gitea
Actions](https://blog.gitea.io/2022/12/feature-preview-gitea-actions/)

---------

Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: ChristopherHX <christopher.homberger@web.de>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2023-01-31 09:45:19 +08:00

417 lines
13 KiB
Go

// Copyright 2020 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package repository
import (
"context"
"errors"
"fmt"
"strings"
"time"
"code.gitea.io/gitea/models/db"
git_model "code.gitea.io/gitea/models/git"
repo_model "code.gitea.io/gitea/models/repo"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/cache"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/graceful"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/notification"
"code.gitea.io/gitea/modules/process"
"code.gitea.io/gitea/modules/queue"
repo_module "code.gitea.io/gitea/modules/repository"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/timeutil"
issue_service "code.gitea.io/gitea/services/issue"
pull_service "code.gitea.io/gitea/services/pull"
)
// pushQueue represents a queue to handle update pull request tests
var pushQueue queue.Queue
// handle passed PR IDs and test the PRs
func handle(data ...queue.Data) []queue.Data {
for _, datum := range data {
opts := datum.([]*repo_module.PushUpdateOptions)
if err := pushUpdates(opts); err != nil {
log.Error("pushUpdate failed: %v", err)
}
}
return nil
}
func initPushQueue() error {
pushQueue = queue.CreateQueue("push_update", handle, []*repo_module.PushUpdateOptions{})
if pushQueue == nil {
return errors.New("unable to create push_update Queue")
}
go graceful.GetManager().RunWithShutdownFns(pushQueue.Run)
return nil
}
// PushUpdate is an alias of PushUpdates for single push update options
func PushUpdate(opts *repo_module.PushUpdateOptions) error {
return PushUpdates([]*repo_module.PushUpdateOptions{opts})
}
// PushUpdates adds a push update to push queue
func PushUpdates(opts []*repo_module.PushUpdateOptions) error {
if len(opts) == 0 {
return nil
}
for _, opt := range opts {
if opt.IsNewRef() && opt.IsDelRef() {
return fmt.Errorf("Old and new revisions are both %s", git.EmptySHA)
}
}
return pushQueue.Push(opts)
}
// pushUpdates generates push action history feeds for push updating multiple refs
func pushUpdates(optsList []*repo_module.PushUpdateOptions) error {
if len(optsList) == 0 {
return nil
}
ctx, _, finished := process.GetManager().AddContext(graceful.GetManager().HammerContext(), fmt.Sprintf("PushUpdates: %s/%s", optsList[0].RepoUserName, optsList[0].RepoName))
defer finished()
repo, err := repo_model.GetRepositoryByOwnerAndName(ctx, optsList[0].RepoUserName, optsList[0].RepoName)
if err != nil {
return fmt.Errorf("GetRepositoryByOwnerAndName failed: %w", err)
}
repoPath := repo.RepoPath()
gitRepo, err := git.OpenRepository(ctx, repoPath)
if err != nil {
return fmt.Errorf("OpenRepository[%s]: %w", repoPath, err)
}
defer gitRepo.Close()
if err = repo_module.UpdateRepoSize(ctx, repo); err != nil {
log.Error("Failed to update size for repository: %v", err)
}
addTags := make([]string, 0, len(optsList))
delTags := make([]string, 0, len(optsList))
var pusher *user_model.User
for _, opts := range optsList {
log.Trace("pushUpdates: %-v %s %s %s", repo, opts.OldCommitID, opts.NewCommitID, opts.RefFullName)
if opts.IsNewRef() && opts.IsDelRef() {
return fmt.Errorf("old and new revisions are both %s", git.EmptySHA)
}
if opts.IsTag() { // If is tag reference
if pusher == nil || pusher.ID != opts.PusherID {
if opts.PusherID == user_model.ActionsUserID {
pusher = user_model.NewActionsUser()
} else {
var err error
if pusher, err = user_model.GetUserByID(ctx, opts.PusherID); err != nil {
return err
}
}
}
tagName := opts.TagName()
if opts.IsDelRef() {
notification.NotifyPushCommits(
db.DefaultContext, pusher, repo,
&repo_module.PushUpdateOptions{
RefFullName: git.TagPrefix + tagName,
OldCommitID: opts.OldCommitID,
NewCommitID: git.EmptySHA,
}, repo_module.NewPushCommits())
delTags = append(delTags, tagName)
notification.NotifyDeleteRef(db.DefaultContext, pusher, repo, "tag", opts.RefFullName)
} else { // is new tag
newCommit, err := gitRepo.GetCommit(opts.NewCommitID)
if err != nil {
return fmt.Errorf("gitRepo.GetCommit(%s) in %s/%s[%d]: %w", opts.NewCommitID, repo.OwnerName, repo.Name, repo.ID, err)
}
commits := repo_module.NewPushCommits()
commits.HeadCommit = repo_module.CommitToPushCommit(newCommit)
commits.CompareURL = repo.ComposeCompareURL(git.EmptySHA, opts.NewCommitID)
notification.NotifyPushCommits(
db.DefaultContext, pusher, repo,
&repo_module.PushUpdateOptions{
RefFullName: git.TagPrefix + tagName,
OldCommitID: git.EmptySHA,
NewCommitID: opts.NewCommitID,
}, commits)
addTags = append(addTags, tagName)
notification.NotifyCreateRef(db.DefaultContext, pusher, repo, "tag", opts.RefFullName, opts.NewCommitID)
}
} else if opts.IsBranch() { // If is branch reference
if pusher == nil || pusher.ID != opts.PusherID {
if opts.PusherID == user_model.ActionsUserID {
pusher = user_model.NewActionsUser()
} else {
var err error
if pusher, err = user_model.GetUserByID(ctx, opts.PusherID); err != nil {
return err
}
}
}
branch := opts.BranchName()
if !opts.IsDelRef() {
log.Trace("TriggerTask '%s/%s' by %s", repo.Name, branch, pusher.Name)
go pull_service.AddTestPullRequestTask(pusher, repo.ID, branch, true, opts.OldCommitID, opts.NewCommitID)
newCommit, err := gitRepo.GetCommit(opts.NewCommitID)
if err != nil {
return fmt.Errorf("gitRepo.GetCommit(%s) in %s/%s[%d]: %w", opts.NewCommitID, repo.OwnerName, repo.Name, repo.ID, err)
}
refName := opts.RefName()
// Push new branch.
var l []*git.Commit
if opts.IsNewRef() {
if repo.IsEmpty { // Change default branch and empty status only if pushed ref is non-empty branch.
repo.DefaultBranch = refName
repo.IsEmpty = false
if repo.DefaultBranch != setting.Repository.DefaultBranch {
if err := gitRepo.SetDefaultBranch(repo.DefaultBranch); err != nil {
if !git.IsErrUnsupportedVersion(err) {
return err
}
}
}
// Update the is empty and default_branch columns
if err := repo_model.UpdateRepositoryCols(db.DefaultContext, repo, "default_branch", "is_empty"); err != nil {
return fmt.Errorf("UpdateRepositoryCols: %w", err)
}
}
l, err = newCommit.CommitsBeforeLimit(10)
if err != nil {
return fmt.Errorf("newCommit.CommitsBeforeLimit: %w", err)
}
notification.NotifyCreateRef(db.DefaultContext, pusher, repo, "branch", opts.RefFullName, opts.NewCommitID)
} else {
l, err = newCommit.CommitsBeforeUntil(opts.OldCommitID)
if err != nil {
return fmt.Errorf("newCommit.CommitsBeforeUntil: %w", err)
}
isForce, err := repo_module.IsForcePush(ctx, opts)
if err != nil {
log.Error("isForcePush %s:%s failed: %v", repo.FullName(), branch, err)
}
if isForce {
log.Trace("Push %s is a force push", opts.NewCommitID)
cache.Remove(repo.GetCommitsCountCacheKey(opts.RefName(), true))
} else {
// TODO: increment update the commit count cache but not remove
cache.Remove(repo.GetCommitsCountCacheKey(opts.RefName(), true))
}
}
commits := repo_module.GitToPushCommits(l)
commits.HeadCommit = repo_module.CommitToPushCommit(newCommit)
if err := issue_service.UpdateIssuesCommit(pusher, repo, commits.Commits, refName); err != nil {
log.Error("updateIssuesCommit: %v", err)
}
oldCommitID := opts.OldCommitID
if oldCommitID == git.EmptySHA && len(commits.Commits) > 0 {
oldCommit, err := gitRepo.GetCommit(commits.Commits[len(commits.Commits)-1].Sha1)
if err != nil && !git.IsErrNotExist(err) {
log.Error("unable to GetCommit %s from %-v: %v", oldCommitID, repo, err)
}
if oldCommit != nil {
for i := 0; i < oldCommit.ParentCount(); i++ {
commitID, _ := oldCommit.ParentID(i)
if !commitID.IsZero() {
oldCommitID = commitID.String()
break
}
}
}
}
if oldCommitID == git.EmptySHA && repo.DefaultBranch != branch {
oldCommitID = repo.DefaultBranch
}
if oldCommitID != git.EmptySHA {
commits.CompareURL = repo.ComposeCompareURL(oldCommitID, opts.NewCommitID)
} else {
commits.CompareURL = ""
}
if len(commits.Commits) > setting.UI.FeedMaxCommitNum {
commits.Commits = commits.Commits[:setting.UI.FeedMaxCommitNum]
}
notification.NotifyPushCommits(db.DefaultContext, pusher, repo, opts, commits)
if err = git_model.RemoveDeletedBranchByName(ctx, repo.ID, branch); err != nil {
log.Error("models.RemoveDeletedBranch %s/%s failed: %v", repo.ID, branch, err)
}
// Cache for big repository
if err := CacheRef(graceful.GetManager().HammerContext(), repo, gitRepo, opts.RefFullName); err != nil {
log.Error("repo_module.CacheRef %s/%s failed: %v", repo.ID, branch, err)
}
} else {
notification.NotifyDeleteRef(db.DefaultContext, pusher, repo, "branch", opts.RefFullName)
if err = pull_service.CloseBranchPulls(pusher, repo.ID, branch); err != nil {
// close all related pulls
log.Error("close related pull request failed: %v", err)
}
}
// Even if user delete a branch on a repository which he didn't watch, he will be watch that.
if err = repo_model.WatchIfAuto(db.DefaultContext, opts.PusherID, repo.ID, true); err != nil {
log.Warn("Fail to perform auto watch on user %v for repo %v: %v", opts.PusherID, repo.ID, err)
}
} else {
log.Trace("Non-tag and non-branch commits pushed.")
}
}
if err := PushUpdateAddDeleteTags(repo, gitRepo, addTags, delTags); err != nil {
return fmt.Errorf("PushUpdateAddDeleteTags: %w", err)
}
// Change repository last updated time.
if err := repo_model.UpdateRepositoryUpdatedTime(repo.ID, time.Now()); err != nil {
return fmt.Errorf("UpdateRepositoryUpdatedTime: %w", err)
}
return nil
}
// PushUpdateAddDeleteTags updates a number of added and delete tags
func PushUpdateAddDeleteTags(repo *repo_model.Repository, gitRepo *git.Repository, addTags, delTags []string) error {
return db.WithTx(db.DefaultContext, func(ctx context.Context) error {
if err := repo_model.PushUpdateDeleteTagsContext(ctx, repo, delTags); err != nil {
return err
}
return pushUpdateAddTags(ctx, repo, gitRepo, addTags)
})
}
// pushUpdateAddTags updates a number of add tags
func pushUpdateAddTags(ctx context.Context, repo *repo_model.Repository, gitRepo *git.Repository, tags []string) error {
if len(tags) == 0 {
return nil
}
lowerTags := make([]string, 0, len(tags))
for _, tag := range tags {
lowerTags = append(lowerTags, strings.ToLower(tag))
}
releases, err := repo_model.GetReleasesByRepoIDAndNames(ctx, repo.ID, lowerTags)
if err != nil {
return fmt.Errorf("GetReleasesByRepoIDAndNames: %w", err)
}
relMap := make(map[string]*repo_model.Release)
for _, rel := range releases {
relMap[rel.LowerTagName] = rel
}
newReleases := make([]*repo_model.Release, 0, len(lowerTags)-len(relMap))
emailToUser := make(map[string]*user_model.User)
for i, lowerTag := range lowerTags {
tag, err := gitRepo.GetTag(tags[i])
if err != nil {
return fmt.Errorf("GetTag: %w", err)
}
commit, err := tag.Commit(gitRepo)
if err != nil {
return fmt.Errorf("Commit: %w", err)
}
sig := tag.Tagger
if sig == nil {
sig = commit.Author
}
if sig == nil {
sig = commit.Committer
}
var author *user_model.User
createdAt := time.Unix(1, 0)
if sig != nil {
var ok bool
author, ok = emailToUser[sig.Email]
if !ok {
author, err = user_model.GetUserByEmailContext(ctx, sig.Email)
if err != nil && !user_model.IsErrUserNotExist(err) {
return fmt.Errorf("GetUserByEmail: %w", err)
}
if author != nil {
emailToUser[sig.Email] = author
}
}
createdAt = sig.When
}
commitsCount, err := commit.CommitsCount()
if err != nil {
return fmt.Errorf("CommitsCount: %w", err)
}
rel, has := relMap[lowerTag]
if !has {
rel = &repo_model.Release{
RepoID: repo.ID,
Title: "",
TagName: tags[i],
LowerTagName: lowerTag,
Target: "",
Sha1: commit.ID.String(),
NumCommits: commitsCount,
Note: "",
IsDraft: false,
IsPrerelease: false,
IsTag: true,
CreatedUnix: timeutil.TimeStamp(createdAt.Unix()),
}
if author != nil {
rel.PublisherID = author.ID
}
newReleases = append(newReleases, rel)
} else {
rel.Sha1 = commit.ID.String()
rel.CreatedUnix = timeutil.TimeStamp(createdAt.Unix())
rel.NumCommits = commitsCount
rel.IsDraft = false
if rel.IsTag && author != nil {
rel.PublisherID = author.ID
}
if err = repo_model.UpdateRelease(ctx, rel); err != nil {
return fmt.Errorf("Update: %w", err)
}
}
}
if len(newReleases) > 0 {
if err = db.Insert(ctx, newReleases); err != nil {
return fmt.Errorf("Insert: %w", err)
}
}
return nil
}