[dpdk-stable] [RFC stable-scripts] 1-import: don't filter commits for later releases that fix backported patches

[dpdk-stable] [RFC stable-scripts] 1-import: don't filter commits for later releases that fix backported patches

[Luca Boccassi]

A patch might fix a commit in a later release and so it is skipped. But
that commit might have been backported in the past, so skipping the
patch is wrong.
Rework the list pruning to take this case into account.

Signed-off-by: Luca Boccassi <bluca@debian.org>
---
Tested and seems to be working fine. Slows down by a few seconds the
runtime, but it's bearable and I'm sure it can be optimised, but it's
a starting point.
More testing would be very welcome.

 1-import | 39 ++++++++++++++++++++++++++++-----------
 1 file changed, 28 insertions(+), 11 deletions(-)

diff --git a/1-import b/1-import
index 05dfdf0..9f03c53 100755
--- a/1-import
+++ b/1-import
@@ -28,18 +28,35 @@ import_one_commit()
 if [ "$PRUNE_COMMITS" = "yes" ]
 then
 	get_stable_release
-	# need to update it in 2021!
-	for _major in {16..20}
+	tmp_list=$PWD/$stable_release/tmp_list
+	rm -f $tmp_list
+
+	pushd $STABLE_DIR &>/dev/null
+	log=$($GIT log v${RTE_VER_MAJOR}..v$last_release)
+	while read line
 	do
-		for _minor in {02..11..3}
-		do
-			if verlte $stable_release $_major.$_minor
-			then
-				sed -i "/($_major.$_minor (partially.*\$/d" $commit_list
-				sed -i "/($_major.$_minor)\$/d" $commit_list
-			fi
-		done
-	done
+		# Get the id of the commit being fixed, for the first check
+		id=$(echo $line | awk '{print $2}')
+		fixes=$($GIT log --format='%b' -1 $id | sed -n 's,^ *\([Ff]ixes\|[Rr]everts\): *\([0-9a-f]*\).*,\2,p')
+
+		# Was stable CC'ed without a Fixes line? If so select it just in case
+		if echo $line | grep -q "(N/A)"
+		then
+			echo $line >> $tmp_list
+		# Is the patch fixing a commit that was part of this stable release?
+		# If so select it
+		elif verlte `echo $line | sed "s/.*(\([0-9][0-9].[0-9][0-9]\).*)/\1/"` $stable_release
+		then
+			echo $line >> $tmp_list
+		# Was the commit (which might be only in a subsequent release)
+		# that the patch "fixes" backported? If so select it
+		elif test -n fixes && echo $log | grep -q -e "backported from commit $fixes" -e "upstream commit $fixes"
+		then
+			echo $line >> $tmp_list
+		fi
+	done < $commit_list
+	popd &>/dev/null
+	mv $tmp_list $commit_list
 fi
 
 if [ "$GIT_AM_PAUSE_ON_FAIL" = "yes" ]
-- 
2.18.0

[dpdk-stable] [RFC stable-scripts v2] 1-import: don't filter commits for later releases that fix backported patches

[Luca Boccassi]

A patch might fix a commit in a later release and so it is skipped. But
that commit might have been backported in the past, so skipping the
patch is wrong.
Rework the list pruning to take this case into account.

Signed-off-by: Luca Boccassi <bluca@debian.org>
---
v2: create $stable_release directory in case it's not already there
    (first round of backporting)

 1-import | 40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)

diff --git a/1-import b/1-import
index 05dfdf0..497c0ae 100755
--- a/1-import
+++ b/1-import
@@ -28,18 +28,36 @@ import_one_commit()
 if [ "$PRUNE_COMMITS" = "yes" ]
 then
 	get_stable_release
-	# need to update it in 2021!
-	for _major in {16..20}
+	tmp_list=$PWD/$stable_release/tmp_list
+	rm -f $tmp_list
+	mkdir -p $PWD/$stable_release
+
+	pushd $STABLE_DIR &>/dev/null
+	log=$($GIT log v${RTE_VER_MAJOR}..v$last_release)
+	while read line
 	do
-		for _minor in {02..11..3}
-		do
-			if verlte $stable_release $_major.$_minor
-			then
-				sed -i "/($_major.$_minor (partially.*\$/d" $commit_list
-				sed -i "/($_major.$_minor)\$/d" $commit_list
-			fi
-		done
-	done
+		# Get the id of the commit being fixed, for the first check
+		id=$(echo $line | awk '{print $2}')
+		fixes=$($GIT log --format='%b' -1 $id | sed -n 's,^ *\([Ff]ixes\|[Rr]everts\): *\([0-9a-f]*\).*,\2,p')
+
+		# Was stable CC'ed without a Fixes line? If so select it just in case
+		if echo $line | grep -q "(N/A)"
+		then
+			echo $line >> $tmp_list
+		# Is the patch fixing a commit that was part of this stable release?
+		# If so select it
+		elif verlte `echo $line | sed "s/.*(\([0-9][0-9].[0-9][0-9]\).*)/\1/"` $stable_release
+		then
+			echo $line >> $tmp_list
+		# Was the commit (which might be only in a subsequent release)
+		# that the patch "fixes" backported? If so select it
+		elif test -n fixes && echo $log | grep -q -e "backported from commit $fixes" -e "upstream commit $fixes"
+		then
+			echo $line >> $tmp_list
+		fi
+	done < $commit_list
+	popd &>/dev/null
+	mv $tmp_list $commit_list
 fi
 
 if [ "$GIT_AM_PAUSE_ON_FAIL" = "yes" ]
-- 
2.18.0

Re: [dpdk-stable] [RFC stable-scripts v2] 1-import: don't filter commits for later releases that fix backported patches

[Luca Boccassi]

On Tue, 2018-09-25 at 13:36 +0100, Luca Boccassi wrote:
> A patch might fix a commit in a later release and so it is skipped.
> But
> that commit might have been backported in the past, so skipping the
> patch is wrong.
> Rework the list pruning to take this case into account.
> 
> Signed-off-by: Luca Boccassi <bluca@debian.org>
> ---
> v2: create $stable_release directory in case it's not already there
>     (first round of backporting)
> 
>  1-import | 40 +++++++++++++++++++++++++++++-----------
>  1 file changed, 29 insertions(+), 11 deletions(-)
> 
> diff --git a/1-import b/1-import
> index 05dfdf0..497c0ae 100755
> --- a/1-import
> +++ b/1-import
> @@ -28,18 +28,36 @@ import_one_commit()
>  if [ "$PRUNE_COMMITS" = "yes" ]
>  then
>  	get_stable_release
> -	# need to update it in 2021!
> -	for _major in {16..20}
> +	tmp_list=$PWD/$stable_release/tmp_list
> +	rm -f $tmp_list
> +	mkdir -p $PWD/$stable_release
> +
> +	pushd $STABLE_DIR &>/dev/null
> +	log=$($GIT log v${RTE_VER_MAJOR}..v$last_release)
> +	while read line
>  	do
> -		for _minor in {02..11..3}
> -		do
> -			if verlte $stable_release $_major.$_minor
> -			then
> -				sed -i "/($_major.$_minor
> (partially.*\$/d" $commit_list
> -				sed -i "/($_major.$_minor)\$/d"
> $commit_list
> -			fi
> -		done
> -	done
> +		# Get the id of the commit being fixed, for the
> first check
> +		id=$(echo $line | awk '{print $2}')
> +		fixes=$($GIT log --format='%b' -1 $id | sed -n 's,^
> *\([Ff]ixes\|[Rr]everts\): *\([0-9a-f]*\).*,\2,p')
> +
> +		# Was stable CC'ed without a Fixes line? If so
> select it just in case
> +		if echo $line | grep -q "(N/A)"
> +		then
> +			echo $line >> $tmp_list
> +		# Is the patch fixing a commit that was part of this
> stable release?
> +		# If so select it
> +		elif verlte `echo $line | sed "s/.*(\([0-9][0-9].[0-
> 9][0-9]\).*)/\1/"` $stable_release
> +		then
> +			echo $line >> $tmp_list
> +		# Was the commit (which might be only in a
> subsequent release)
> +		# that the patch "fixes" backported? If so select it
> +		elif test -n fixes && echo $log | grep -q -e
> "backported from commit $fixes" -e "upstream commit $fixes"
> +		then
> +			echo $line >> $tmp_list
> +		fi
> +	done < $commit_list
> +	popd &>/dev/null
> +	mv $tmp_list $commit_list
>  fi
>  
>  if [ "$GIT_AM_PAUSE_ON_FAIL" = "yes" ]

Ping - any chance of a review?

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-stable] [RFC stable-scripts v2] 1-import: don't filter commits for later releases that fix backported patches

[Kevin Traynor]

On 09/25/2018 01:36 PM, Luca Boccassi wrote:
> A patch might fix a commit in a later release and so it is skipped. But
> that commit might have been backported in the past, so skipping the
> patch is wrong.

Just a minor thing, but I was a bit confused by the first sentence
above. It was clear when I read the code/comment. How about rewording to
something like: A patch might fix a commit that has already been
backported to the stable branch but is not yet part of a stable release.


Acked-by: Kevin Traynor <ktraynor@redhat.com>

> Rework the list pruning to take this case into account.
> 
> Signed-off-by: Luca Boccassi <bluca@debian.org>
> ---
> v2: create $stable_release directory in case it's not already there
>     (first round of backporting)
> 
>  1-import | 40 +++++++++++++++++++++++++++++-----------
>  1 file changed, 29 insertions(+), 11 deletions(-)
> 
> diff --git a/1-import b/1-import
> index 05dfdf0..497c0ae 100755
> --- a/1-import
> +++ b/1-import
> @@ -28,18 +28,36 @@ import_one_commit()
>  if [ "$PRUNE_COMMITS" = "yes" ]
>  then
>  	get_stable_release
> -	# need to update it in 2021!
> -	for _major in {16..20}
> +	tmp_list=$PWD/$stable_release/tmp_list
> +	rm -f $tmp_list
> +	mkdir -p $PWD/$stable_release
> +
> +	pushd $STABLE_DIR &>/dev/null
> +	log=$($GIT log v${RTE_VER_MAJOR}..v$last_release)
> +	while read line
>  	do
> -		for _minor in {02..11..3}
> -		do
> -			if verlte $stable_release $_major.$_minor
> -			then
> -				sed -i "/($_major.$_minor (partially.*\$/d" $commit_list
> -				sed -i "/($_major.$_minor)\$/d" $commit_list
> -			fi
> -		done
> -	done
> +		# Get the id of the commit being fixed, for the first check
> +		id=$(echo $line | awk '{print $2}')
> +		fixes=$($GIT log --format='%b' -1 $id | sed -n 's,^ *\([Ff]ixes\|[Rr]everts\): *\([0-9a-f]*\).*,\2,p')
> +
> +		# Was stable CC'ed without a Fixes line? If so select it just in case
> +		if echo $line | grep -q "(N/A)"
> +		then
> +			echo $line >> $tmp_list
> +		# Is the patch fixing a commit that was part of this stable release?
> +		# If so select it
> +		elif verlte `echo $line | sed "s/.*(\([0-9][0-9].[0-9][0-9]\).*)/\1/"` $stable_release
> +		then
> +			echo $line >> $tmp_list
> +		# Was the commit (which might be only in a subsequent release)
> +		# that the patch "fixes" backported? If so select it
> +		elif test -n fixes && echo $log | grep -q -e "backported from commit $fixes" -e "upstream commit $fixes"
> +		then
> +			echo $line >> $tmp_list
> +		fi
> +	done < $commit_list
> +	popd &>/dev/null
> +	mv $tmp_list $commit_list
>  fi
>  
>  if [ "$GIT_AM_PAUSE_ON_FAIL" = "yes" ]
> 

Re: [dpdk-stable] [RFC stable-scripts v2] 1-import: don't filter commits for later releases that fix backported patches

[Luca Boccassi]

On Mon, 2018-10-08 at 11:58 +0100, Kevin Traynor wrote:
> On 09/25/2018 01:36 PM, Luca Boccassi wrote:
> > A patch might fix a commit in a later release and so it is skipped.
> > But
> > that commit might have been backported in the past, so skipping the
> > patch is wrong.
> 
> Just a minor thing, but I was a bit confused by the first sentence
> above. It was clear when I read the code/comment. How about rewording
> to
> something like: A patch might fix a commit that has already been
> backported to the stable branch but is not yet part of a stable
> release.
> 
> 
> Acked-by: Kevin Traynor <ktraynor@redhat.com>

Thanks, changed and pushed

-- 
Kind regards,
Luca Boccassi